Find the answer to your Linux question:
Results 1 to 6 of 6
Hello. Briefly the problem is slowness of copying a big file into flashdisk. It is a new HP v210w 32 gb flashdisk. It is FAT32 formatted. When I try to ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Dec 2012
    Posts
    2

    Weird cache flush problem with 32 gb flashdisk


    Hello. Briefly the problem is slowness of copying a big file into flashdisk.

    It is a new HP v210w 32 gb flashdisk. It is FAT32 formatted. When I try to write a big file (bigger than about 50 MB) into it, after writing about half of data, it begins to struggle.

    An example, it is a 321 MB file, 4th coloumn is write speed in KB/sec, as one line per second appended to output:

    Code:
    > iostat -k -t 1 |grep sdc
    sdc               0,06         1,24         0,00       8661          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              66,00         0,00      7618,50          0       7618
    sdc              84,00         0,00     10008,00          0      10008
    sdc              85,00         0,00      9792,50          0       9792
    sdc              84,00         0,00     10008,00          0      10008
    sdc              83,00         0,00      9960,00          0       9960
    sdc              82,00         0,00      9840,00          0       9840
    sdc              83,00         0,00      9960,00          0       9960
    sdc              85,00         0,00      9665,00          0       9665
    sdc              83,00         0,00      9960,00          0       9960
    sdc              81,00         0,00      9696,00          0       9696
    sdc              81,00         0,00      9720,00          0       9720
    sdc              85,00         0,00      9702,00          0       9702
    sdc              79,00         0,00      9480,00          0       9480
    sdc              93,00         0,00      9836,00          0       9836
    sdc              44,00         0,00      4436,00          0       4436
    sdc               0,00         0,00         0,00          0          0
    sdc              26,00         0,00      2532,00          0       2532
    sdc              81,00         0,00      7744,00          0       7744
    sdc               0,00         0,00         0,00          0          0
    sdc               8,00         0,00       820,00          0        820
    sdc              92,00         0,00      9195,50          0       9195
    sdc              16,00         0,00      1508,00          0       1508
    sdc               1,00         0,00        24,00          0         24
    sdc              28,00         0,00      2820,00          0       2820
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              25,00         0,00      2400,00          0       2400
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              78,00         0,00      7556,00          0       7556
    sdc              48,00         0,00      4856,00          0       4856
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              35,00         0,00      3472,50          0       3472
    sdc               1,00         0,00        20,00          0         20
    sdc              13,00         0,00      1284,00          0       1284
    sdc               0,00         0,00         0,00          0          0
    sdc               5,00         0,00       448,00          0        448
    sdc              54,00         0,00      5364,00          0       5364
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              12,00         0,00      1176,00          0       1176
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              44,00         0,00      4340,00          0       4340
    sdc               0,00         0,00         0,00          0          0
    sdc               3,00         0,00       348,00          0        348
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              41,00         0,00      4068,00          0       4068
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              32,00         0,00      3276,00          0       3276
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              63,00         0,00      6400,00          0       6400
    sdc              45,00         0,00      4392,00          0       4392
    sdc               0,00         0,00         0,00          0          0
    sdc              22,00         0,00      1956,00          0       1956
    sdc              53,00         0,00      5320,00          0       5320
    sdc               0,00         0,00         0,00          0          0
    sdc              64,00         0,00      6156,00          0       6156
    sdc              14,00         0,00      1480,00          0       1480
    sdc               0,00         0,00         0,00          0          0
    sdc               8,00         0,00       724,00          0        724
    sdc              19,00         0,00      1920,00          0       1920
    sdc              23,00         0,00      2492,00          0       2492
    sdc              43,00         0,00      4812,00          0       4812
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               9,00         0,00       760,00          0        760
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              13,00         0,00      1368,00          0       1368
    sdc               0,00         0,00         0,00          0          0
    sdc               1,00         0,00        52,00          0         52
    sdc               3,00         0,00       232,00          0        232
    sdc               0,00         0,00         0,00          0          0
    sdc               3,00         0,00       356,00          0        356
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              14,00         0,00      1248,00          0       1248
    sdc               0,00         0,00         0,00          0          0
    sdc              67,00         0,00      6700,00          0       6700
    sdc              16,00         0,00      1604,00          0       1604
    sdc               0,00         0,00         0,00          0          0
    sdc               6,00         0,00       508,00          0        508
    sdc              37,00         0,00      3764,00          0       3764
    sdc              11,00         0,00       916,00          0        916
    sdc              91,00         0,00      9388,00          0       9388
    sdc              82,00         0,00      8556,00          0       8556
    sdc               0,00         0,00         0,00          0          0
    sdc               1,00         0,00        76,00          0         76
    sdc               7,00         0,00       768,00          0        768
    sdc               0,00         0,00         0,00          0          0
    sdc              11,00         0,00       952,00          0        952
    sdc              51,00         0,00      4856,00          0       4856
    sdc               1,00         0,00       108,00          0        108
    sdc              86,00         0,00      8652,00          0       8652
    sdc              27,00         0,00      2996,00          0       2996
    sdc              16,00         0,00      1484,00          0       1484
    sdc               3,00         0,00       360,00          0        360
    sdc               0,00         0,00         0,00          0          0
    sdc              11,00         0,00       980,00          0        980
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               4,00         0,00       384,00          0        384
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              52,00         0,00      5284,00          0       5284
    sdc              49,00         0,00      5008,00          0       5008
    sdc               0,00         0,00         0,00          0          0
    sdc              82,00         0,00      7832,00          0       7832
    sdc               7,00         0,00       744,00          0        744
    sdc               0,00         0,00         0,00          0          0
    sdc              16,00         0,00      1528,00          0       1528
    sdc              97,00         0,00      9652,00          0       9652
    sdc              29,00         0,00      3284,50          0       3284
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    No problem for 390 MB of 98 files:

    Code:
    > iostat -k -t 1 |grep sdc
    sdc               0,49         1,19        44,90       8726     328987
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc              56,00         0,00      5955,00          0       5955
    sdc               0,00         0,00         0,00          0          0
    sdc               2,00         0,00       160,00          0        160
    sdc              37,00         0,00      4360,00          0       4360
    sdc              84,00         0,00      9872,00          0       9872
    sdc              84,16         0,00      9308,91          0       9402
    sdc              84,00         0,00      9822,50          0       9822
    sdc              83,00         0,00      9219,00          0       9219
    sdc              81,00         0,00      9609,00          0       9609
    sdc              87,00         0,00      9205,50          0       9205
    sdc              82,00         0,00      9587,00          0       9587
    sdc              85,00         0,00      9403,50          0       9403
    sdc              89,00         0,00      9074,50          0       9074
    sdc              82,00         0,00      9768,50          0       9768
    sdc              84,00         0,00     10004,50          0      10004
    sdc              87,00         0,00      9307,00          0       9307
    sdc              80,00         0,00      9543,50          0       9543
    sdc              89,00         0,00      9366,50          0       9366
    sdc               6,00         0,00       574,50          0        574
    sdc               0,00         0,00         0,00          0          0
    sdc              38,00         0,00      4508,00          0       4508
    sdc              84,00         0,00      9080,50          0       9080
    sdc              16,00         0,00      1920,00          0       1920
    sdc              79,00         0,00      9280,00          0       9280
    sdc              84,00         0,00      9261,50          0       9261
    sdc              82,00         0,00      9714,00          0       9714
    sdc              81,00         0,00      9072,00          0       9072
    sdc              80,00         0,00      9345,00          0       9345
    sdc              82,00         0,00      9840,00          0       9840
    sdc              84,00         0,00      8949,50          0       8949
    sdc              81,00         0,00      9556,50          0       9556
    sdc              83,00         0,00      8665,50          0       8665
    sdc              82,00         0,00      9749,50          0       9749
    sdc              83,00         0,00      9944,50          0       9944
    sdc              85,00         0,00      8827,50          0       8827
    sdc              81,00         0,00      9491,50          0       9491
    sdc              85,00         0,00      9136,50          0       9136
    sdc              77,00         0,00      9238,50          0       9238
    sdc              83,00         0,00      9058,00          0       9058
    sdc              80,00         0,00      9170,00          0       9170
    sdc              83,00         0,00      8861,50          0       8861
    sdc              83,00         0,00      9908,00          0       9908
    sdc              86,00         0,00      9056,00          0       9056
    sdc              82,00         0,00      9590,50          0       9590
    sdc              82,00         0,00      8719,50          0       8719
    sdc              82,00         0,00      9644,50          0       9644
    sdc              83,00         0,00      8997,50          0       8997
    sdc              78,00         0,00      9314,00          0       9314
    sdc              72,00         0,00      7935,50          0       7935
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    sdc               0,00         0,00         0,00          0          0
    Also no problem when I write the file with "dd" with "sync" flag.

    I use OpenSuse 12.2, kernel is 3.4.11-2.16.

    Mount options of disk:
    /dev/sdc1 on /media/BALIKHAFIZA type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=100,fmask=0 022,dmask=0077,codepage=cp437,iocharset=iso8859-1,shortname=mixed,showexec,utf8,flush,errors=remou nt-ro,uhelper=udisks)
    Last edited by Behzat; 12-16-2012 at 04:01 PM.

  2. #2
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,677
    This is not an uncommon problem, which is why I don't use USB flash drives, but class 10 SD cards with a USB carrier. These are the (currently) highest performing flash drives available for write performance. Many, small files will normally work ok (a lot of that is cached with write-behind behavior), but a few really big files will force the system to write directly to disc. As for "dd" with "sync" flag, that will write each sector to disc as it is passed to the operating system, so it will utilize the hardware bandwidth more effectively. As an experiment, if you were to write all those small files with something like "cp", when it was done, try executing the "sync" command (as root), and see how long before the data is REALLY written to the device! A lot longer than you expect, I think.

    So, get the fastest flash devices you can if write performance is critical - that means using class 10 SD cards and NOT normal USB flash drives.
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  3. #3
    Super Moderator devils casper's Avatar
    Join Date
    Jun 2006
    Location
    Chandigarh, India
    Posts
    24,729
    Have you tried using default options in mount command?
    Code:
    mount -t vfat /dev/sdc1  /media/BALIKHAFIZA -o  defaults,umask=0
    It is amazing what you can accomplish if you do not care who gets the credit.
    New Users: Read This First

  4. $spacer_open
    $spacer_close
  5. #4
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,677
    Quote Originally Posted by devils casper View Post
    Have you tried using default options in mount command?
    Code:
    mount -t vfat /dev/sdc1  /media/BALIKHAFIZA -o  defaults,umask=0
    Interesting DC. Can you explain what you mean about this? What do these default options change about how the flash drive will behave?
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  6. #5
    Just Joined!
    Join Date
    Dec 2012
    Posts
    2
    Quote Originally Posted by Rubberman View Post
    This is not an uncommon problem, which is why I don't use USB flash drives, but class 10 SD cards with a USB carrier. These are the (currently) highest performing flash drives available for write performance. Many, small files will normally work ok (a lot of that is cached with write-behind behavior), but a few really big files will force the system to write directly to disc. As for "dd" with "sync" flag, that will write each sector to disc as it is passed to the operating system, so it will utilize the hardware bandwidth more effectively. As an experiment, if you were to write all those small files with something like "cp", when it was done, try executing the "sync" command (as root), and see how long before the data is REALLY written to the device! A lot longer than you expect, I think.

    So, get the fastest flash devices you can if write performance is critical - that means using class 10 SD cards and NOT normal USB flash drives.
    But they tend to be fragile and comlicated, so not for my pocket Indeed I tried them before. They have their problems, harder to solve. For example I couldn't boot computer. I couldn't use it on satelite receiver's video recorder. It is a featured high quality usb carrier. I think it is hard to find a carrier which emulates with full features.

    Quote Originally Posted by devils casper View Post
    Have you tried using default options in mount command?
    Code:
    mount -t vfat /dev/sdc1  /media/BALIKHAFIZA -o  defaults,umask=0
    It worked The problem is the "flush" option. Adding it to options repeats the issue.

    Before this topic I suspected of it in the first place and experimented with options without "flush" but it behaved with the same problem. Or somehow it fooled me.

    Thanks

  7. #6
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,677
    Quote Originally Posted by Behzat View Post
    But they tend to be fragile and comlicated, so not for my pocket Indeed I tried them before. They have their problems, harder to solve. For example I couldn't boot computer. I couldn't use it on satelite receiver's video recorder. It is a featured high quality usb carrier. I think it is hard to find a carrier which emulates with full features.



    It worked The problem is the "flush" option. Adding it to options repeats the issue.

    Before this topic I suspected of it in the first place and experimented with options without "flush" but it behaved with the same problem. Or somehow it fooled me.

    Thanks
    Glad you got this sorted out. FWIW, I use SD and micro-SD cards in USB carriers all the time (they look like thumb drives, but with a slot in the back for the SD card) and can install and boot/run Linux on them without any trouble. In fact, I have a little old chocolate tin that I keep a bunch in, each with a different live distribution installed, so I can easily demonstrate a number of distributions to people or in classes without any trouble at all. IE, if I am teaching a class that is server related, I boot the CentOS one. If I am teaching a class that is related to general user services, then I boot an Ubuntu or Mint one. I just use a Sharpie to mark them so I know which is which...
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •