Is there a way to handle large arrays of colums from MySQL when importing via $POST









up vote
0
down vote

favorite












I am trying to work on a API to import data to MySQL Database.
I have it working on a small scale.



I have now altered it so its more in line with the current database. Its a large table and been poorly structured, But for simplicity and maintaining am trying to keep MySQL database and the Applications database inline.
MSQL = Windows based application:
MySQL = The Server database, Am extracting Data out of MSQL hosted locally and importing it to a MySQL Database hosted externally.



As i have expanded the code, its no longer working, this could be a number of factors like the table needs structuring / indexing / primary, or i am passing too much data via a query. The application is starting to look very messy.



The question is, is it necessary and better practice me listing out each column and matching up the data sets apposed to trying to utilize the MySQL INFORMATION_SCHEMA to list the columns instead.



Here is the code so far, I have already removed a number of columns.



We have a Server and Host:



Server Code



Class file: Api.php



class API

private $connect = '';

function __construct()

$this->database_connection();


function database_connection()

$this->connect = new PDO("mysql:host=$Host;dbname=$DatabaseName", $DatabaseUsername, $DatabasePassword);

function insert() // Insert Function

if(isset($_POST["Branch"]))

$form_data = array(
':Branch'=>$_POST['Branch'],
':Date'=>$_POST['Date'],
':Week'=>$_POST['Week'],
':Period'=>$_POST['Period'],
':Invoice_No'=>$_POST['Invoice_No'],
':Invoice_Reference'=>$_POST['Invoice_Reference'],
':Line_No'=>$_POST['Line_No'],
':Till_No'=>$_POST['Till_No'],
':Operator'=>$_POST['Operator'],
':Stock_Code'=>$_POST['Stock_Code'],
':Barcode'=>$_POST['Barcode'],
':Line_Quantity'=>$_POST['Line_Quantity'],
':Weight'=>$_POST['Weight'],
':Weight_Unit'=>$_POST['Weight_Unit'],
':Man_Weighed'=>$_POST['Man_Weighed'],
':Unit_ID'=>$_POST['Unit_ID'],
':Line_Price_Band'=>$_POST['Line_Price_Band'],
':Original_Sell'=>$_POST['Original_Sell'],
':Actual_Sell'=>$_POST['Actual_Sell'],
':Cost'=>$_POST['Cost'],
':Vat_Rate'=>$_POST['Vat_Rate'],
':Discount_Rate'=>$_POST['Discount_Rate'],
':Value_Goods'=>$_POST['Value_Goods'],
':Value_VAT'=>$_POST['Value_VAT'],
':Value_Sale'=>$_POST['Value_Sale'],
':Value_Cost'=>$_POST['Value_Cost'],
':Price_Override_Amount'=>$_POST['Price_Override_Amount'],
':Price_Overrided'=>$_POST['Price_Overrided'],
':Discounted'=>$_POST['Discounted'],
':Account_No'=>$_POST['Account_No'],
':Sub_Account_No'=>$_POST['Sub_Account_No'],
':Customer_Account'=>$_POST['Customer_Account'],
':Sub_Customer_Account'=>$_POST['Sub_Customer_Account'],
':Cust_Type_ID'=>$_POST['Cust_Type_ID'],
':Super_Department_ID'=>$_POST['Super_Department_ID'],
':Department_ID'=>$_POST['Department_ID'],
':Group_ID'=>$_POST['Group_ID'],
':Sub_Group_ID'=>$_POST['Sub_Group_ID'],
':Retail_Location_ID'=>$_POST['Retail_Location_ID'],
':Retail_Sub_Location_ID'=>$_POST['Retail_Sub_Location_ID'],
':Entry_Method'=>$_POST['Entry_Method'],
':End_Sale_Discount_Rate'=>$_POST['End_Sale_Discount_Rate'],
':Loyalty_Discount_Rate'=>$_POST['Loyalty_Discount_Rate'],
':Date_Time_Stamp'=>$_POST['Date_Time_Stamp']

);
$query = "
INSERT INTO Lines
(Branch, Date, Week, Period, Invoice_No, Invoice_Reference, Line_No, Till_No, Operator, Stock_Code, Barcode, Line_Quantity, Weight, Weight_Unit, Man_Weighed, Unit_ID, Line_Price_Band, Original_Sell, Actual_Sell, Cost, Vat_Rate, Discount_Rate, Value_Goods, Value_VAT, Value_Sale, Value_Cost, Price_Override_Amount, Price_Overrided, Discounted, Account_No, Sub_Account_No, Customer_Account, Sub_Customer_Account, Cust_Type_ID, Super_Department_ID, Department_ID, Group_ID, Sub_Group_ID, Retail_Location_ID, Retail_Sub_Location_ID, Entry_Method, End_Sale_Discount_Rate, Loyalty_Discount_Rate, Date_Time_Stamp) VALUES
(:Branch, :Date, :Week, :Period, :Invoice_No, :Invoice_Reference, :Line_No, :Till_No, :Operator, :Stock_Code, :Barcode, :Line_Quantity, :Weight, :Weight_Unit, :Man_Weighed, :Unit_ID, :Line_Price_Band, :Original_Sell, :Actual_Sell, :Cost, :Vat_Rate, :Discount_Rate, :Value_Goods, :Value_VAT, :Value_Sale, :Value_Cost, :Price_Override_Amount, :Price_Overrided, :Discounted, :Account_No, :Sub_Account_No, :Customer_Account, :Sub_Customer_Account, :Cust_Type_ID, :Super_Department_ID, :Department_ID, :Group_ID, :Sub_Group_ID, :Retail_Location_ID, :Retail_Sub_Location_ID, :Entry_Method, :End_Sale_Discount_Rate, :Loyalty_Discount_Rate, :Date_Time_Stamp)
";
$statement = $this->connect->prepare($query);
if($statement->execute($form_data))

echo $statement ;
$data = array(
'success' => '1'
);

else

var_dump($statement) ;
$data = array(
'success' => '0'
);


else

$data = array(
'success' => '0'
);

return $data;




Calling the Function: Handler test_api.php
// This does have an include for Api.php



if($_GET["action"] == 'insert')

$data = $api_object->insert();



Host code



This wont be the final thing, as ill be extracting from MSQL building the array and posting this way, This is just laid out like so for testing



<?php

$form_data = array(
'Branch' => '1',
'Date' => '8/11/2018 13:42:00',
'Week' => '1',
'Period' => '1',
'Invoice_No' => '9999998',
'Invoice_Reference' => '99999998',
'Line_No' => '1',
'Till_No' => '1',
'Operator' => '99',
'Stock_Code' => '123456',
'Barcode' => '654321',
'Line_Quantity' => '99',
'Weight' => '',
'Weight_Unit' => '',
'Man_Weighed' => '',
'Unit_ID' => '',
'Line_Price_Band' => '1',
'Original_Sell' => '99.99',
'Actual_Sell' => '99.99',
'Cost' => '9.99',
'Vat_Rate' => '1',
'Discount_Rate' => '',
'Value_Goods' => '',
'Value_VAT' => '',
'Value_Sale' => '',
'Value_Cost' => '',
'Price_Override_Amount' => '',
'Price_Overrided' => '',
'Discounted' => '',
'Account_No' => '',
'Sub_Account_No' => '',
'Customer_Account' => '',
'Sub_Customer_Account' => '',
'Cust_Type_ID' => '',
'Super_Department_ID' => '',
'Department_ID' => '',
'Group_ID' => '',
'Sub_Group_ID' => '',
'Retail_Location_ID' => '',
'Retail_Sub_Location_ID' => '',
'Entry_Method' => '',
'End_Sale_Discount_Rate' => '',
'Loyalty_Discount_Rate' => '',
'Date_Time_Stamp' => ''

);
$api_url = "localhost/Modules/API/Server/test_api.php?action=insert"; //change this url as per your folder path for api folder
$client = curl_init($api_url);
curl_setopt($client, CURLOPT_POST, true);
curl_setopt($client, CURLOPT_POSTFIELDS, $form_data);
curl_setopt($client, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($client);
// var_dump($response);
curl_close($client);

$result = json_decode($response, true);
echo $response ;
foreach($result as $keys => $values)

if($result[$keys]['success'] == '1')

echo 'update';

else

echo 'error';
echo $response ;




At the moment its not working, I am not sure why, As it was working fine with a much smaller dataset.
And my IDE hasnt reported of any errors or typos, But thats something else, if you do spot a error, please do say.



The question is, is there a better way to work with large number of columns.



In previous API's i have basically just dumped the data to txt and imported from a deliminator. So this is all very new to me.










share|improve this question























  • AFAIK Curl doesn't support posting arrays - stackoverflow.com/questions/13596799/…
    – Nigel Ren
    Nov 8 at 14:20






  • 3




    It looks like you are sending one row per request. That is very costly. I would recommend to send multiple rows per request (e.g. 200). Also I would look into sending the data as JSON Body, instead of using classic form post.
    – Erik Kalkoken
    Nov 8 at 14:27











  • Excuse me for asking, but is there any reason why you do not import the SQLs directly or via the mysql import tools?
    – Erik Kalkoken
    Nov 8 at 14:35










  • Control, The end product would go down to around 100 clients. I didn't want to have to have Cron Jobs or files floating around and being processed, Importing directly would have to open up remote connections on any IP address. In previous experience with uploading files and internet drop outs corrupted files, lock key. i thought this process would remove a ton of issues i have had previously.
    – Dave Hamilton
    Nov 8 at 14:39














up vote
0
down vote

favorite












I am trying to work on a API to import data to MySQL Database.
I have it working on a small scale.



I have now altered it so its more in line with the current database. Its a large table and been poorly structured, But for simplicity and maintaining am trying to keep MySQL database and the Applications database inline.
MSQL = Windows based application:
MySQL = The Server database, Am extracting Data out of MSQL hosted locally and importing it to a MySQL Database hosted externally.



As i have expanded the code, its no longer working, this could be a number of factors like the table needs structuring / indexing / primary, or i am passing too much data via a query. The application is starting to look very messy.



The question is, is it necessary and better practice me listing out each column and matching up the data sets apposed to trying to utilize the MySQL INFORMATION_SCHEMA to list the columns instead.



Here is the code so far, I have already removed a number of columns.



We have a Server and Host:



Server Code



Class file: Api.php



class API

private $connect = '';

function __construct()

$this->database_connection();


function database_connection()

$this->connect = new PDO("mysql:host=$Host;dbname=$DatabaseName", $DatabaseUsername, $DatabasePassword);

function insert() // Insert Function

if(isset($_POST["Branch"]))

$form_data = array(
':Branch'=>$_POST['Branch'],
':Date'=>$_POST['Date'],
':Week'=>$_POST['Week'],
':Period'=>$_POST['Period'],
':Invoice_No'=>$_POST['Invoice_No'],
':Invoice_Reference'=>$_POST['Invoice_Reference'],
':Line_No'=>$_POST['Line_No'],
':Till_No'=>$_POST['Till_No'],
':Operator'=>$_POST['Operator'],
':Stock_Code'=>$_POST['Stock_Code'],
':Barcode'=>$_POST['Barcode'],
':Line_Quantity'=>$_POST['Line_Quantity'],
':Weight'=>$_POST['Weight'],
':Weight_Unit'=>$_POST['Weight_Unit'],
':Man_Weighed'=>$_POST['Man_Weighed'],
':Unit_ID'=>$_POST['Unit_ID'],
':Line_Price_Band'=>$_POST['Line_Price_Band'],
':Original_Sell'=>$_POST['Original_Sell'],
':Actual_Sell'=>$_POST['Actual_Sell'],
':Cost'=>$_POST['Cost'],
':Vat_Rate'=>$_POST['Vat_Rate'],
':Discount_Rate'=>$_POST['Discount_Rate'],
':Value_Goods'=>$_POST['Value_Goods'],
':Value_VAT'=>$_POST['Value_VAT'],
':Value_Sale'=>$_POST['Value_Sale'],
':Value_Cost'=>$_POST['Value_Cost'],
':Price_Override_Amount'=>$_POST['Price_Override_Amount'],
':Price_Overrided'=>$_POST['Price_Overrided'],
':Discounted'=>$_POST['Discounted'],
':Account_No'=>$_POST['Account_No'],
':Sub_Account_No'=>$_POST['Sub_Account_No'],
':Customer_Account'=>$_POST['Customer_Account'],
':Sub_Customer_Account'=>$_POST['Sub_Customer_Account'],
':Cust_Type_ID'=>$_POST['Cust_Type_ID'],
':Super_Department_ID'=>$_POST['Super_Department_ID'],
':Department_ID'=>$_POST['Department_ID'],
':Group_ID'=>$_POST['Group_ID'],
':Sub_Group_ID'=>$_POST['Sub_Group_ID'],
':Retail_Location_ID'=>$_POST['Retail_Location_ID'],
':Retail_Sub_Location_ID'=>$_POST['Retail_Sub_Location_ID'],
':Entry_Method'=>$_POST['Entry_Method'],
':End_Sale_Discount_Rate'=>$_POST['End_Sale_Discount_Rate'],
':Loyalty_Discount_Rate'=>$_POST['Loyalty_Discount_Rate'],
':Date_Time_Stamp'=>$_POST['Date_Time_Stamp']

);
$query = "
INSERT INTO Lines
(Branch, Date, Week, Period, Invoice_No, Invoice_Reference, Line_No, Till_No, Operator, Stock_Code, Barcode, Line_Quantity, Weight, Weight_Unit, Man_Weighed, Unit_ID, Line_Price_Band, Original_Sell, Actual_Sell, Cost, Vat_Rate, Discount_Rate, Value_Goods, Value_VAT, Value_Sale, Value_Cost, Price_Override_Amount, Price_Overrided, Discounted, Account_No, Sub_Account_No, Customer_Account, Sub_Customer_Account, Cust_Type_ID, Super_Department_ID, Department_ID, Group_ID, Sub_Group_ID, Retail_Location_ID, Retail_Sub_Location_ID, Entry_Method, End_Sale_Discount_Rate, Loyalty_Discount_Rate, Date_Time_Stamp) VALUES
(:Branch, :Date, :Week, :Period, :Invoice_No, :Invoice_Reference, :Line_No, :Till_No, :Operator, :Stock_Code, :Barcode, :Line_Quantity, :Weight, :Weight_Unit, :Man_Weighed, :Unit_ID, :Line_Price_Band, :Original_Sell, :Actual_Sell, :Cost, :Vat_Rate, :Discount_Rate, :Value_Goods, :Value_VAT, :Value_Sale, :Value_Cost, :Price_Override_Amount, :Price_Overrided, :Discounted, :Account_No, :Sub_Account_No, :Customer_Account, :Sub_Customer_Account, :Cust_Type_ID, :Super_Department_ID, :Department_ID, :Group_ID, :Sub_Group_ID, :Retail_Location_ID, :Retail_Sub_Location_ID, :Entry_Method, :End_Sale_Discount_Rate, :Loyalty_Discount_Rate, :Date_Time_Stamp)
";
$statement = $this->connect->prepare($query);
if($statement->execute($form_data))

echo $statement ;
$data = array(
'success' => '1'
);

else

var_dump($statement) ;
$data = array(
'success' => '0'
);


else

$data = array(
'success' => '0'
);

return $data;




Calling the Function: Handler test_api.php
// This does have an include for Api.php



if($_GET["action"] == 'insert')

$data = $api_object->insert();



Host code



This wont be the final thing, as ill be extracting from MSQL building the array and posting this way, This is just laid out like so for testing



<?php

$form_data = array(
'Branch' => '1',
'Date' => '8/11/2018 13:42:00',
'Week' => '1',
'Period' => '1',
'Invoice_No' => '9999998',
'Invoice_Reference' => '99999998',
'Line_No' => '1',
'Till_No' => '1',
'Operator' => '99',
'Stock_Code' => '123456',
'Barcode' => '654321',
'Line_Quantity' => '99',
'Weight' => '',
'Weight_Unit' => '',
'Man_Weighed' => '',
'Unit_ID' => '',
'Line_Price_Band' => '1',
'Original_Sell' => '99.99',
'Actual_Sell' => '99.99',
'Cost' => '9.99',
'Vat_Rate' => '1',
'Discount_Rate' => '',
'Value_Goods' => '',
'Value_VAT' => '',
'Value_Sale' => '',
'Value_Cost' => '',
'Price_Override_Amount' => '',
'Price_Overrided' => '',
'Discounted' => '',
'Account_No' => '',
'Sub_Account_No' => '',
'Customer_Account' => '',
'Sub_Customer_Account' => '',
'Cust_Type_ID' => '',
'Super_Department_ID' => '',
'Department_ID' => '',
'Group_ID' => '',
'Sub_Group_ID' => '',
'Retail_Location_ID' => '',
'Retail_Sub_Location_ID' => '',
'Entry_Method' => '',
'End_Sale_Discount_Rate' => '',
'Loyalty_Discount_Rate' => '',
'Date_Time_Stamp' => ''

);
$api_url = "localhost/Modules/API/Server/test_api.php?action=insert"; //change this url as per your folder path for api folder
$client = curl_init($api_url);
curl_setopt($client, CURLOPT_POST, true);
curl_setopt($client, CURLOPT_POSTFIELDS, $form_data);
curl_setopt($client, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($client);
// var_dump($response);
curl_close($client);

$result = json_decode($response, true);
echo $response ;
foreach($result as $keys => $values)

if($result[$keys]['success'] == '1')

echo 'update';

else

echo 'error';
echo $response ;




At the moment its not working, I am not sure why, As it was working fine with a much smaller dataset.
And my IDE hasnt reported of any errors or typos, But thats something else, if you do spot a error, please do say.



The question is, is there a better way to work with large number of columns.



In previous API's i have basically just dumped the data to txt and imported from a deliminator. So this is all very new to me.










share|improve this question























  • AFAIK Curl doesn't support posting arrays - stackoverflow.com/questions/13596799/…
    – Nigel Ren
    Nov 8 at 14:20






  • 3




    It looks like you are sending one row per request. That is very costly. I would recommend to send multiple rows per request (e.g. 200). Also I would look into sending the data as JSON Body, instead of using classic form post.
    – Erik Kalkoken
    Nov 8 at 14:27











  • Excuse me for asking, but is there any reason why you do not import the SQLs directly or via the mysql import tools?
    – Erik Kalkoken
    Nov 8 at 14:35










  • Control, The end product would go down to around 100 clients. I didn't want to have to have Cron Jobs or files floating around and being processed, Importing directly would have to open up remote connections on any IP address. In previous experience with uploading files and internet drop outs corrupted files, lock key. i thought this process would remove a ton of issues i have had previously.
    – Dave Hamilton
    Nov 8 at 14:39












up vote
0
down vote

favorite









up vote
0
down vote

favorite











I am trying to work on a API to import data to MySQL Database.
I have it working on a small scale.



I have now altered it so its more in line with the current database. Its a large table and been poorly structured, But for simplicity and maintaining am trying to keep MySQL database and the Applications database inline.
MSQL = Windows based application:
MySQL = The Server database, Am extracting Data out of MSQL hosted locally and importing it to a MySQL Database hosted externally.



As i have expanded the code, its no longer working, this could be a number of factors like the table needs structuring / indexing / primary, or i am passing too much data via a query. The application is starting to look very messy.



The question is, is it necessary and better practice me listing out each column and matching up the data sets apposed to trying to utilize the MySQL INFORMATION_SCHEMA to list the columns instead.



Here is the code so far, I have already removed a number of columns.



We have a Server and Host:



Server Code



Class file: Api.php



class API

private $connect = '';

function __construct()

$this->database_connection();


function database_connection()

$this->connect = new PDO("mysql:host=$Host;dbname=$DatabaseName", $DatabaseUsername, $DatabasePassword);

function insert() // Insert Function

if(isset($_POST["Branch"]))

$form_data = array(
':Branch'=>$_POST['Branch'],
':Date'=>$_POST['Date'],
':Week'=>$_POST['Week'],
':Period'=>$_POST['Period'],
':Invoice_No'=>$_POST['Invoice_No'],
':Invoice_Reference'=>$_POST['Invoice_Reference'],
':Line_No'=>$_POST['Line_No'],
':Till_No'=>$_POST['Till_No'],
':Operator'=>$_POST['Operator'],
':Stock_Code'=>$_POST['Stock_Code'],
':Barcode'=>$_POST['Barcode'],
':Line_Quantity'=>$_POST['Line_Quantity'],
':Weight'=>$_POST['Weight'],
':Weight_Unit'=>$_POST['Weight_Unit'],
':Man_Weighed'=>$_POST['Man_Weighed'],
':Unit_ID'=>$_POST['Unit_ID'],
':Line_Price_Band'=>$_POST['Line_Price_Band'],
':Original_Sell'=>$_POST['Original_Sell'],
':Actual_Sell'=>$_POST['Actual_Sell'],
':Cost'=>$_POST['Cost'],
':Vat_Rate'=>$_POST['Vat_Rate'],
':Discount_Rate'=>$_POST['Discount_Rate'],
':Value_Goods'=>$_POST['Value_Goods'],
':Value_VAT'=>$_POST['Value_VAT'],
':Value_Sale'=>$_POST['Value_Sale'],
':Value_Cost'=>$_POST['Value_Cost'],
':Price_Override_Amount'=>$_POST['Price_Override_Amount'],
':Price_Overrided'=>$_POST['Price_Overrided'],
':Discounted'=>$_POST['Discounted'],
':Account_No'=>$_POST['Account_No'],
':Sub_Account_No'=>$_POST['Sub_Account_No'],
':Customer_Account'=>$_POST['Customer_Account'],
':Sub_Customer_Account'=>$_POST['Sub_Customer_Account'],
':Cust_Type_ID'=>$_POST['Cust_Type_ID'],
':Super_Department_ID'=>$_POST['Super_Department_ID'],
':Department_ID'=>$_POST['Department_ID'],
':Group_ID'=>$_POST['Group_ID'],
':Sub_Group_ID'=>$_POST['Sub_Group_ID'],
':Retail_Location_ID'=>$_POST['Retail_Location_ID'],
':Retail_Sub_Location_ID'=>$_POST['Retail_Sub_Location_ID'],
':Entry_Method'=>$_POST['Entry_Method'],
':End_Sale_Discount_Rate'=>$_POST['End_Sale_Discount_Rate'],
':Loyalty_Discount_Rate'=>$_POST['Loyalty_Discount_Rate'],
':Date_Time_Stamp'=>$_POST['Date_Time_Stamp']

);
$query = "
INSERT INTO Lines
(Branch, Date, Week, Period, Invoice_No, Invoice_Reference, Line_No, Till_No, Operator, Stock_Code, Barcode, Line_Quantity, Weight, Weight_Unit, Man_Weighed, Unit_ID, Line_Price_Band, Original_Sell, Actual_Sell, Cost, Vat_Rate, Discount_Rate, Value_Goods, Value_VAT, Value_Sale, Value_Cost, Price_Override_Amount, Price_Overrided, Discounted, Account_No, Sub_Account_No, Customer_Account, Sub_Customer_Account, Cust_Type_ID, Super_Department_ID, Department_ID, Group_ID, Sub_Group_ID, Retail_Location_ID, Retail_Sub_Location_ID, Entry_Method, End_Sale_Discount_Rate, Loyalty_Discount_Rate, Date_Time_Stamp) VALUES
(:Branch, :Date, :Week, :Period, :Invoice_No, :Invoice_Reference, :Line_No, :Till_No, :Operator, :Stock_Code, :Barcode, :Line_Quantity, :Weight, :Weight_Unit, :Man_Weighed, :Unit_ID, :Line_Price_Band, :Original_Sell, :Actual_Sell, :Cost, :Vat_Rate, :Discount_Rate, :Value_Goods, :Value_VAT, :Value_Sale, :Value_Cost, :Price_Override_Amount, :Price_Overrided, :Discounted, :Account_No, :Sub_Account_No, :Customer_Account, :Sub_Customer_Account, :Cust_Type_ID, :Super_Department_ID, :Department_ID, :Group_ID, :Sub_Group_ID, :Retail_Location_ID, :Retail_Sub_Location_ID, :Entry_Method, :End_Sale_Discount_Rate, :Loyalty_Discount_Rate, :Date_Time_Stamp)
";
$statement = $this->connect->prepare($query);
if($statement->execute($form_data))

echo $statement ;
$data = array(
'success' => '1'
);

else

var_dump($statement) ;
$data = array(
'success' => '0'
);


else

$data = array(
'success' => '0'
);

return $data;




Calling the Function: Handler test_api.php
// This does have an include for Api.php



if($_GET["action"] == 'insert')

$data = $api_object->insert();



Host code



This wont be the final thing, as ill be extracting from MSQL building the array and posting this way, This is just laid out like so for testing



<?php

$form_data = array(
'Branch' => '1',
'Date' => '8/11/2018 13:42:00',
'Week' => '1',
'Period' => '1',
'Invoice_No' => '9999998',
'Invoice_Reference' => '99999998',
'Line_No' => '1',
'Till_No' => '1',
'Operator' => '99',
'Stock_Code' => '123456',
'Barcode' => '654321',
'Line_Quantity' => '99',
'Weight' => '',
'Weight_Unit' => '',
'Man_Weighed' => '',
'Unit_ID' => '',
'Line_Price_Band' => '1',
'Original_Sell' => '99.99',
'Actual_Sell' => '99.99',
'Cost' => '9.99',
'Vat_Rate' => '1',
'Discount_Rate' => '',
'Value_Goods' => '',
'Value_VAT' => '',
'Value_Sale' => '',
'Value_Cost' => '',
'Price_Override_Amount' => '',
'Price_Overrided' => '',
'Discounted' => '',
'Account_No' => '',
'Sub_Account_No' => '',
'Customer_Account' => '',
'Sub_Customer_Account' => '',
'Cust_Type_ID' => '',
'Super_Department_ID' => '',
'Department_ID' => '',
'Group_ID' => '',
'Sub_Group_ID' => '',
'Retail_Location_ID' => '',
'Retail_Sub_Location_ID' => '',
'Entry_Method' => '',
'End_Sale_Discount_Rate' => '',
'Loyalty_Discount_Rate' => '',
'Date_Time_Stamp' => ''

);
$api_url = "localhost/Modules/API/Server/test_api.php?action=insert"; //change this url as per your folder path for api folder
$client = curl_init($api_url);
curl_setopt($client, CURLOPT_POST, true);
curl_setopt($client, CURLOPT_POSTFIELDS, $form_data);
curl_setopt($client, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($client);
// var_dump($response);
curl_close($client);

$result = json_decode($response, true);
echo $response ;
foreach($result as $keys => $values)

if($result[$keys]['success'] == '1')

echo 'update';

else

echo 'error';
echo $response ;




At the moment its not working, I am not sure why, As it was working fine with a much smaller dataset.
And my IDE hasnt reported of any errors or typos, But thats something else, if you do spot a error, please do say.



The question is, is there a better way to work with large number of columns.



In previous API's i have basically just dumped the data to txt and imported from a deliminator. So this is all very new to me.










share|improve this question















I am trying to work on a API to import data to MySQL Database.
I have it working on a small scale.



I have now altered it so its more in line with the current database. Its a large table and been poorly structured, But for simplicity and maintaining am trying to keep MySQL database and the Applications database inline.
MSQL = Windows based application:
MySQL = The Server database, Am extracting Data out of MSQL hosted locally and importing it to a MySQL Database hosted externally.



As i have expanded the code, its no longer working, this could be a number of factors like the table needs structuring / indexing / primary, or i am passing too much data via a query. The application is starting to look very messy.



The question is, is it necessary and better practice me listing out each column and matching up the data sets apposed to trying to utilize the MySQL INFORMATION_SCHEMA to list the columns instead.



Here is the code so far, I have already removed a number of columns.



We have a Server and Host:



Server Code



Class file: Api.php



class API

private $connect = '';

function __construct()

$this->database_connection();


function database_connection()

$this->connect = new PDO("mysql:host=$Host;dbname=$DatabaseName", $DatabaseUsername, $DatabasePassword);

function insert() // Insert Function

if(isset($_POST["Branch"]))

$form_data = array(
':Branch'=>$_POST['Branch'],
':Date'=>$_POST['Date'],
':Week'=>$_POST['Week'],
':Period'=>$_POST['Period'],
':Invoice_No'=>$_POST['Invoice_No'],
':Invoice_Reference'=>$_POST['Invoice_Reference'],
':Line_No'=>$_POST['Line_No'],
':Till_No'=>$_POST['Till_No'],
':Operator'=>$_POST['Operator'],
':Stock_Code'=>$_POST['Stock_Code'],
':Barcode'=>$_POST['Barcode'],
':Line_Quantity'=>$_POST['Line_Quantity'],
':Weight'=>$_POST['Weight'],
':Weight_Unit'=>$_POST['Weight_Unit'],
':Man_Weighed'=>$_POST['Man_Weighed'],
':Unit_ID'=>$_POST['Unit_ID'],
':Line_Price_Band'=>$_POST['Line_Price_Band'],
':Original_Sell'=>$_POST['Original_Sell'],
':Actual_Sell'=>$_POST['Actual_Sell'],
':Cost'=>$_POST['Cost'],
':Vat_Rate'=>$_POST['Vat_Rate'],
':Discount_Rate'=>$_POST['Discount_Rate'],
':Value_Goods'=>$_POST['Value_Goods'],
':Value_VAT'=>$_POST['Value_VAT'],
':Value_Sale'=>$_POST['Value_Sale'],
':Value_Cost'=>$_POST['Value_Cost'],
':Price_Override_Amount'=>$_POST['Price_Override_Amount'],
':Price_Overrided'=>$_POST['Price_Overrided'],
':Discounted'=>$_POST['Discounted'],
':Account_No'=>$_POST['Account_No'],
':Sub_Account_No'=>$_POST['Sub_Account_No'],
':Customer_Account'=>$_POST['Customer_Account'],
':Sub_Customer_Account'=>$_POST['Sub_Customer_Account'],
':Cust_Type_ID'=>$_POST['Cust_Type_ID'],
':Super_Department_ID'=>$_POST['Super_Department_ID'],
':Department_ID'=>$_POST['Department_ID'],
':Group_ID'=>$_POST['Group_ID'],
':Sub_Group_ID'=>$_POST['Sub_Group_ID'],
':Retail_Location_ID'=>$_POST['Retail_Location_ID'],
':Retail_Sub_Location_ID'=>$_POST['Retail_Sub_Location_ID'],
':Entry_Method'=>$_POST['Entry_Method'],
':End_Sale_Discount_Rate'=>$_POST['End_Sale_Discount_Rate'],
':Loyalty_Discount_Rate'=>$_POST['Loyalty_Discount_Rate'],
':Date_Time_Stamp'=>$_POST['Date_Time_Stamp']

);
$query = "
INSERT INTO Lines
(Branch, Date, Week, Period, Invoice_No, Invoice_Reference, Line_No, Till_No, Operator, Stock_Code, Barcode, Line_Quantity, Weight, Weight_Unit, Man_Weighed, Unit_ID, Line_Price_Band, Original_Sell, Actual_Sell, Cost, Vat_Rate, Discount_Rate, Value_Goods, Value_VAT, Value_Sale, Value_Cost, Price_Override_Amount, Price_Overrided, Discounted, Account_No, Sub_Account_No, Customer_Account, Sub_Customer_Account, Cust_Type_ID, Super_Department_ID, Department_ID, Group_ID, Sub_Group_ID, Retail_Location_ID, Retail_Sub_Location_ID, Entry_Method, End_Sale_Discount_Rate, Loyalty_Discount_Rate, Date_Time_Stamp) VALUES
(:Branch, :Date, :Week, :Period, :Invoice_No, :Invoice_Reference, :Line_No, :Till_No, :Operator, :Stock_Code, :Barcode, :Line_Quantity, :Weight, :Weight_Unit, :Man_Weighed, :Unit_ID, :Line_Price_Band, :Original_Sell, :Actual_Sell, :Cost, :Vat_Rate, :Discount_Rate, :Value_Goods, :Value_VAT, :Value_Sale, :Value_Cost, :Price_Override_Amount, :Price_Overrided, :Discounted, :Account_No, :Sub_Account_No, :Customer_Account, :Sub_Customer_Account, :Cust_Type_ID, :Super_Department_ID, :Department_ID, :Group_ID, :Sub_Group_ID, :Retail_Location_ID, :Retail_Sub_Location_ID, :Entry_Method, :End_Sale_Discount_Rate, :Loyalty_Discount_Rate, :Date_Time_Stamp)
";
$statement = $this->connect->prepare($query);
if($statement->execute($form_data))

echo $statement ;
$data = array(
'success' => '1'
);

else

var_dump($statement) ;
$data = array(
'success' => '0'
);


else

$data = array(
'success' => '0'
);

return $data;




Calling the Function: Handler test_api.php
// This does have an include for Api.php



if($_GET["action"] == 'insert')

$data = $api_object->insert();



Host code



This wont be the final thing, as ill be extracting from MSQL building the array and posting this way, This is just laid out like so for testing



<?php

$form_data = array(
'Branch' => '1',
'Date' => '8/11/2018 13:42:00',
'Week' => '1',
'Period' => '1',
'Invoice_No' => '9999998',
'Invoice_Reference' => '99999998',
'Line_No' => '1',
'Till_No' => '1',
'Operator' => '99',
'Stock_Code' => '123456',
'Barcode' => '654321',
'Line_Quantity' => '99',
'Weight' => '',
'Weight_Unit' => '',
'Man_Weighed' => '',
'Unit_ID' => '',
'Line_Price_Band' => '1',
'Original_Sell' => '99.99',
'Actual_Sell' => '99.99',
'Cost' => '9.99',
'Vat_Rate' => '1',
'Discount_Rate' => '',
'Value_Goods' => '',
'Value_VAT' => '',
'Value_Sale' => '',
'Value_Cost' => '',
'Price_Override_Amount' => '',
'Price_Overrided' => '',
'Discounted' => '',
'Account_No' => '',
'Sub_Account_No' => '',
'Customer_Account' => '',
'Sub_Customer_Account' => '',
'Cust_Type_ID' => '',
'Super_Department_ID' => '',
'Department_ID' => '',
'Group_ID' => '',
'Sub_Group_ID' => '',
'Retail_Location_ID' => '',
'Retail_Sub_Location_ID' => '',
'Entry_Method' => '',
'End_Sale_Discount_Rate' => '',
'Loyalty_Discount_Rate' => '',
'Date_Time_Stamp' => ''

);
$api_url = "localhost/Modules/API/Server/test_api.php?action=insert"; //change this url as per your folder path for api folder
$client = curl_init($api_url);
curl_setopt($client, CURLOPT_POST, true);
curl_setopt($client, CURLOPT_POSTFIELDS, $form_data);
curl_setopt($client, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($client);
// var_dump($response);
curl_close($client);

$result = json_decode($response, true);
echo $response ;
foreach($result as $keys => $values)

if($result[$keys]['success'] == '1')

echo 'update';

else

echo 'error';
echo $response ;




At the moment its not working, I am not sure why, As it was working fine with a much smaller dataset.
And my IDE hasnt reported of any errors or typos, But thats something else, if you do spot a error, please do say.



The question is, is there a better way to work with large number of columns.



In previous API's i have basically just dumped the data to txt and imported from a deliminator. So this is all very new to me.







php mysql






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 8 at 14:32









Erik Kalkoken

11.4k32145




11.4k32145










asked Nov 8 at 14:17









Dave Hamilton

3841310




3841310











  • AFAIK Curl doesn't support posting arrays - stackoverflow.com/questions/13596799/…
    – Nigel Ren
    Nov 8 at 14:20






  • 3




    It looks like you are sending one row per request. That is very costly. I would recommend to send multiple rows per request (e.g. 200). Also I would look into sending the data as JSON Body, instead of using classic form post.
    – Erik Kalkoken
    Nov 8 at 14:27











  • Excuse me for asking, but is there any reason why you do not import the SQLs directly or via the mysql import tools?
    – Erik Kalkoken
    Nov 8 at 14:35










  • Control, The end product would go down to around 100 clients. I didn't want to have to have Cron Jobs or files floating around and being processed, Importing directly would have to open up remote connections on any IP address. In previous experience with uploading files and internet drop outs corrupted files, lock key. i thought this process would remove a ton of issues i have had previously.
    – Dave Hamilton
    Nov 8 at 14:39
















  • AFAIK Curl doesn't support posting arrays - stackoverflow.com/questions/13596799/…
    – Nigel Ren
    Nov 8 at 14:20






  • 3




    It looks like you are sending one row per request. That is very costly. I would recommend to send multiple rows per request (e.g. 200). Also I would look into sending the data as JSON Body, instead of using classic form post.
    – Erik Kalkoken
    Nov 8 at 14:27











  • Excuse me for asking, but is there any reason why you do not import the SQLs directly or via the mysql import tools?
    – Erik Kalkoken
    Nov 8 at 14:35










  • Control, The end product would go down to around 100 clients. I didn't want to have to have Cron Jobs or files floating around and being processed, Importing directly would have to open up remote connections on any IP address. In previous experience with uploading files and internet drop outs corrupted files, lock key. i thought this process would remove a ton of issues i have had previously.
    – Dave Hamilton
    Nov 8 at 14:39















AFAIK Curl doesn't support posting arrays - stackoverflow.com/questions/13596799/…
– Nigel Ren
Nov 8 at 14:20




AFAIK Curl doesn't support posting arrays - stackoverflow.com/questions/13596799/…
– Nigel Ren
Nov 8 at 14:20




3




3




It looks like you are sending one row per request. That is very costly. I would recommend to send multiple rows per request (e.g. 200). Also I would look into sending the data as JSON Body, instead of using classic form post.
– Erik Kalkoken
Nov 8 at 14:27





It looks like you are sending one row per request. That is very costly. I would recommend to send multiple rows per request (e.g. 200). Also I would look into sending the data as JSON Body, instead of using classic form post.
– Erik Kalkoken
Nov 8 at 14:27













Excuse me for asking, but is there any reason why you do not import the SQLs directly or via the mysql import tools?
– Erik Kalkoken
Nov 8 at 14:35




Excuse me for asking, but is there any reason why you do not import the SQLs directly or via the mysql import tools?
– Erik Kalkoken
Nov 8 at 14:35












Control, The end product would go down to around 100 clients. I didn't want to have to have Cron Jobs or files floating around and being processed, Importing directly would have to open up remote connections on any IP address. In previous experience with uploading files and internet drop outs corrupted files, lock key. i thought this process would remove a ton of issues i have had previously.
– Dave Hamilton
Nov 8 at 14:39




Control, The end product would go down to around 100 clients. I didn't want to have to have Cron Jobs or files floating around and being processed, Importing directly would have to open up remote connections on any IP address. In previous experience with uploading files and internet drop outs corrupted files, lock key. i thought this process would remove a ton of issues i have had previously.
– Dave Hamilton
Nov 8 at 14:39

















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53209602%2fis-there-a-way-to-handle-large-arrays-of-colums-from-mysql-when-importing-via-p%23new-answer', 'question_page');

);

Post as a guest



































active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes















 

draft saved


draft discarded















































 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53209602%2fis-there-a-way-to-handle-large-arrays-of-colums-from-mysql-when-importing-via-p%23new-answer', 'question_page');

);

Post as a guest














































































Popular posts from this blog

𛂒𛀶,𛀽𛀑𛂀𛃧𛂓𛀙𛃆𛃑𛃷𛂟𛁡𛀢𛀟𛁤𛂽𛁕𛁪𛂟𛂯,𛁞𛂧𛀴𛁄𛁠𛁼𛂿𛀤 𛂘,𛁺𛂾𛃭𛃭𛃵𛀺,𛂣𛃍𛂖𛃶 𛀸𛃀𛂖𛁶𛁏𛁚 𛂢𛂞 𛁰𛂆𛀔,𛁸𛀽𛁓𛃋𛂇𛃧𛀧𛃣𛂐𛃇,𛂂𛃻𛃲𛁬𛃞𛀧𛃃𛀅 𛂭𛁠𛁡𛃇𛀷𛃓𛁥,𛁙𛁘𛁞𛃸𛁸𛃣𛁜,𛂛,𛃿,𛁯𛂘𛂌𛃛𛁱𛃌𛂈𛂇 𛁊𛃲,𛀕𛃴𛀜 𛀶𛂆𛀶𛃟𛂉𛀣,𛂐𛁞𛁾 𛁷𛂑𛁳𛂯𛀬𛃅,𛃶𛁼

Edmonton

Crossroads (UK TV series)