mysql - Optimizing php command line scripts to process large flat files -

-

For the downvoyee fairies .. I know that php is the wrong language for this ... but I I am working. Given that:

I have a large flat file that I need to process in PHP. I convert flat files to mysql in a normalized database. There are several million lines in the flat file.

I originally tried to use an ORM system while importing the flat file, with a great php memory leak problem with the design, with the design being carefully cured. Even if I make sure that there was enough memory, it could take around 25 days for the script to run on my desktop.

I stripped the overhead and created mysql commands directly to write the script again. I removed the auto information from my design because after that I was essentially recorded as a MyScall, which was the last ID entered to create a connection between the data points. I use a global counter for the database ID instead, and I never do any lookup, just inspect it.

I use the Unix Split command to create a lot of smaller files instead of one big one, because one memory is overhead associated with repeatedly using a file pointer

Using these optimizations (hopefully they help someone else) I found the import script to run in about 6 hours.

I hired a virtual example with 5 times more RAM and 5 times more processor power compared to my desktop and it went at exactly the same speed. The server runs the process but the CPU cycle and RAM are skipped. Maybe the limiting factor is disk speed but I have a lot of RAM, should I try to load memory in some way in the files? Any suggestions for more optimization of PSP command line scripts are processed in large files!

You will not like it but ... it sounds like you have a wrong language Are using If you want to make some big leap in motion then for a compiled language, a port will go to the next stage Compiled languages ​​run much more than the scripting language, so that you will leave your processing time.

In addition, you can dump the data into DB using a command build. A postgraze in one (the dump was loaded? Something like that) will read into the delimited text file in the tab, which corresponds to the column in the column. This will give you the attention of getting a text file in the correct format and then it should spit in DB with an order and it will have to handle the optimization instead of itself.

You have done the right thing to talk to the ORM head on the head, you should not need to split the file because your text file reader should use the buffer internally, so it should not be "wanted" Should, but I'm not a * nix man, so it can be wrong on the front.

We have done something similar to every one by placing the Regex through 20 GB files every morning, maintains the memory hash for the unique record and then puts the new ones in the DB. After this, we spit up 9000+ JS files (this is the slowest part) using Ruby Script for ease. We were importer imports in Ruby as well and the whole thing was 3+ hours, rewriting .net runs the whole process in approximately 30-40 minutes and 20 of them are slow ruby ​​script (it is not conducive that it is no longer favorable Not even though this works well).


Comments

Post a Comment

Popular posts from this blog

email - PHP mail error ... failed to open stream : permission denied -

-
I am trying to send an email using PHP mail. I get the following error: WARNING: Unknown: Failed to open the stream: Permission denied in unknown in line 0 Fatal error: unknown: required to open the required '/ Home Php $ name = $ _REQUEST ['name']; $ Email = $ _REQUEST ['email']; $ Theme = $ _REQUEST ['subject']; $ Message = $ _REQUEST ['message']; Mail ("myemail@myhost.com", "topic title", $ message, "sender: $ email"); Header ("Location: http://www.google.com"); ? & Gt; This message is not a mail error PHP file can not open ask_us.php if ask_us If the .php is your file, then check its permissions (and parent folders).
Read more

c# - ListView onScroll event -

-
I am programming a simple C # application, and I need a Scallow event on the list view. So I inherited the original ListView class created in ListviewEx witch. I found out how to locate the scroll message from WinAPI and I modified the WndProc method. Now I have this WndProc: protected override zero WndProc (Ref message message) {base.WndProc (Ref I); If (m.msg == WM_VSCROLL) {onScroll (This, New EventArgs ()); }} But the problem is, I do not know how to know about scrolling information. This data should be in wParam, but there is a LOWORD macro ++ like in C in C # and I switch to find the criteria like SB_ below, SB_ENDSCROLL, SB_PAGEUP etc. there any way Need to change the LOWORD macro from C #? Or any other way to find the necessary parameters about scrolling? You can define the WParam constants as the following: Private const int WM_HSCROLL = 0x114; Private Constant WM_VSCROLL = 0x115; Private contact int SBHOZZ = 0; Private Consultant SB_VERT = 1; Private Con...
Read more

c - Linux mmap() error -

-
I have a memory map file, out of which I want to parse the content of the buffer. MMAP () achieves success, and I can successfully use fprintf to print the buffer content in the file. However, when I try to access Buffer as an array in my program directly, I get a partition error. Why is this happening? Here's the code: #define PTT_DUMP "/ home / druru / somefile". . Int fd_ptt_dump = Open (PTT_DUMP, O_RDONLY); Structure state struct_ptt_dump; Fstat (fd_ptt_dump, & amp; struct_ptt_dump); Printf ("\ n \ n \ t \ t \ t --- Size of dump =% d ----- \ n \ n", struct_ptt_dump.st_size); Four * membuffer; Four pid_num [100]; Four cycles [100], examples [100], CPI [100]; Int pid_index = 0; Int cycles_index = 0; Int instr_index = 0; Int cpi_index = 0; Int len ​​= (int) struct_ptt_dump.st_size; Int newline_count = 0; Int n = 0; If ((membuffer = mmap (0, struct_ptt_dump.st_size, PROT_READ, MAP_FILE | MAP_PRIVATE, fd_ptt_dump, 0)) == (caddr_t) -1) err_sys ("mm...
Read more