Pagini: [1]   În jos
  Imprimă  
Ajutor Subiect: Transpose  (Citit de 5553 ori)
0 Utilizatori şi 1 Vizitator pe acest subiect.
Cosmin
Echipa infoarena
Nu mai tace
*****

Karma: 351
Deconectat Deconectat

Mesaje: 1.799



Vezi Profilul
« : Octombrie 29, 2013, 02:35:04 »

http://www.infoarena.ro/blog/transpose
Memorat
skipy
Strain
*

Karma: 8
Deconectat Deconectat

Mesaje: 46



Vezi Profilul
« Răspunde #1 : Octombrie 29, 2013, 13:15:26 »

A solution can be obtained by splitting the initial matrix into 100 smaller sub-matrices, using files on disk. Since each sub-matrix will have 1 GB, we can load it in memory, transpose it, then write it back to disk. The final step consists of assembling all 100 sub-matrices in the correct order, into another 100GB file.

This works because:
(A B)T = (AT CT)
(C D)     (BT DT)

(assuming A, B, C and D are square sub-matrices).
Memorat

Cheap VR WoW could destroy modern society...
Cosmin1490
Strain


Karma: 1
Deconectat Deconectat

Mesaje: 17



Vezi Profilul
« Răspunde #2 : Octombrie 29, 2013, 16:27:16 »

Is the file in binary or text format ? It wouldn't be that hard to interchange between them but I am curious.

LE: I'm thinking that with the binary format you could seek in that file, this enables you to even transpose it in place,
without even using any significant memory or extra disk space for even larger files. But this is slow due to all those disk read/write/seek operations.
« Ultima modificare: Octombrie 29, 2013, 16:51:59 de către Balan Radu Cosmin » Memorat
marius21
Strain
*

Karma: -20
Deconectat Deconectat

Mesaje: 27



Vezi Profilul
« Răspunde #3 : Octombrie 29, 2013, 21:35:56 »

Read a few lines down of the matrix and start constructing the rows of the transposed matrix left to right in memory. When the memory fills, dump them to disk in a preallocated file by seeking to the beginning of each row if it's binary, or by writing each row to an individual file then cat-ing them together if it's text. You'll only have sqrt(100G) * 100 = 31600000 seeks to do then if it's binary.
Memorat
Pagini: [1]   În sus
  Imprimă  
 
Schimbă forumul:  

Powered by SMF 1.1.19 | SMF © 2006-2013, Simple Machines