Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 12
Hi all Now I changing and developing some kernel stuffs which is related to Linux FileSystems. My work is just to find the solution for the inconvinence use of the ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jan 2010
    Posts
    4

    Linux USB disk development


    Hi all

    Now I changing and developing some kernel stuffs which is related to Linux FileSystems.

    My work is just to find the solution for the inconvinence use of the USB disk. We have to click safe remove for flush the buffer data to disk before the plug off the USB disk .

    So my aim of the work is just avoid to click the safe remove button and plug off the USB disk straightly. Even if buffer cache data couldn't written to USB disk, then plug in the USB disk again and recover the buffer cache data.

    So anyone has a advice ?

    Sorry for my english. i have a little English

  2. #2
    Just Joined!
    Join Date
    Oct 2007
    Posts
    3

    USB disks.

    ganzorig,
    If you don't flush the buffer, you are playing with fire. You cannot go back and finish the job later if all of the files have not been written correctly; at least it have failed every time I tried that plus, it corrupted some thumb drives.

    What is the deal about not making one click? This is not a shortcut you want to use unless you want to pay the consequences. Keep safe and simple.

    Blessings,

    Jon Piper

  3. #3
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,454
    The only way to make this "safe" is to use unbuffered writes to the USB drive, effectively disabling the write-behind cache so you don't need to sync or unmount the drive before unplugging it. This is still dangerous because something may be writing to the device when you pull it, resulting in a scrambled file system.
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  4. #4
    Just Joined!
    Join Date
    Jan 2010
    Posts
    4
    Thanks for your reply JonPiper and Rubberman

    I understand what u're saying. But actually it is my part of my research work. So anyway i have to research and get some experiment whether it is successful or not.

    Here is my Idea step by step. Please check it out

    1. Let's suppose to write A.txt file to USB external hard disk or flash disk.

    in the linux kernel source code, write system call is invoke the following function flows - sys_write (write, int, fd, __user, buf, size_t, count) -> do_sync_write (file, buf, count, pos) -> generic_file_aio_write(kiocb, iov,1, kiocb.ki_pos) -> generic_file_buffered_write (iocb, iov, m_segs, pos,ppos, count, written) -> generic_performance_write (struct file, iov_iter i, pos)

    and then write operation has divided into 3 separate fields. First block_write_begin allocating and initializing buffer heads for the page, Second - copy the data from the user mode buffer to disk's buffer page. Third - block_write_end marks the underlying buffers as disrty so they are written to disk later.

    So my idea is when the kernel perform the write operations then check that the disk is USB or not, and if it is USB disk then copy that page buffers to the some secure buffer which is save the page buffer. Then if we plug off the USB disk before the flush to the disk, then this secure buffers data will be held until that USB disk plug in again or reach the some certain amount of deadline time. If we plug in the USB disk again then check that USB disk and recover the unwritten block to the USB disk.

    So i am using the one library which is implementing the Atomic action. So i think i can use this library's atomic action routines for the Linux kernel read and write operations. If i could successfully use this maybe write and read operation to the usb disk can be a atomically and i can implement the simple recovery block for this stuff

    But actually i am newbie for Kernels and i am not good at C.

    So anyone has tried like this stuff ever ? If yes please help me

  5. #5
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,454
    Think of the failure scenarios before you do anything. Develop up some finite state machines to describe the flow of control and even processing. What happens if you plug another disc in with similar characteristics, such as disc/partition sizes, partition types, names and UUIDs? This is possible if a disc has been cloned. I'm not saying that your goals are impossible, but you need to determine logically and mathematically that you can deal with all the pernicious failure states, and believe me when I tell you that there are a LOT of them here! Get a good system modeling tool, such as Sparx Enterprise Architect, and model this six ways to Sunday before you begin to write even one line of real code.
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  6. #6
    Linux Newbie unlimitedscolobb's Avatar
    Join Date
    Jan 2008
    Posts
    120
    Quote Originally Posted by ganzorig View Post
    So my idea is when the kernel perform the write operations then check that the disk is USB or not, and if it is USB disk then copy that page buffers to the some secure buffer which is save the page buffer. Then if we plug off the USB disk before the flush to the disk, then this secure buffers data will be held until that USB disk plug in again or reach the some certain amount of deadline time. If we plug in the USB disk again then check that USB disk and recover the unwritten block to the USB disk.
    I completely agree with Rubberman -- trying to save one (infinitesimal) mouse click with such measures might be considered an overkill. How would handle reboots, as another example?

    Another problem is that you cannot be sure when it is completely safe to remove the disk (i.e. the buffers have all been synced), so if one is writing somewhat valuable data, one is bound to explicitly sync the drives anyway, which goes against your goals.

  7. #7
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,454
    Unfortunately, unlike floppy, CD, and DVD drives, there is no physical means to lock the storage device from removal (thumb drive or sd card in the USB port or card reader). So, these devices are inherently unsafe from accidental removal (oops, I tripped over the USB cable that was connecting the drive into the system... never done that, right?), and personally, my professional opinion as an engineer with over 30 years in this field is that you cannot avoid such a failure scenario without hardware support. Unfortunately, no one has designed a physical locking mechanism into USB ports that you can program to secure the physical connection when doing critical operations. Of course, that would increate the cost of USB port devices (ports, hubs, etc) by an order of magnitude, and I suspect that would not be well received by the purchasing public.

    Anyway, all they nay-saying aside, I wish you well in your research. FWIW, research that proves that a problem is NP-hard or that a premise is incorrect is also valuable in extending our domain of knowledge, so go for it! Just don't blindly expect that you will be successful. That will result in nothing but the exploration of blind alleys and a waste of valuable time, the only resource you will never be able to replace.
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  8. #8
    Just Joined!
    Join Date
    Feb 2010
    Posts
    16
    Actually, this is an interesting idea. I think Windows has an option in the USB drive to "Turn off buffer caching", but I've never seen it in Linux at all. On the other hand, not using a buffer would slow things down, as well as wearing out writes faster.

  9. #9
    Just Joined!
    Join Date
    Jan 2010
    Posts
    4
    Thanks for replies and your great valuable advices.

    I fully understand what did u guys mentioned. Of course i am not sure that i can complete this work. But once it is my part of research work so i have to do something and have to reach some results.

    I think , initially i should restrict the failure assumptions such that reboot and plug in another disk. So first assumed that there is a only single USB or removal disk plug in the computer and no reboot operation will perform. Also if plug off the disk before the actual write operation to the disk, then have to plug in the same disk again .

    So now my basic idea is check that the disk whether that is removal (USB) or not before the write operation. My question is how can i recognize that disk is USB or not. When the USB disk is plug in , how can i get the block_device structures and magic numbers ?

    I am really sorry for my english.

    Regards

  10. #10
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,454
    Well, just to be a bit more encouraging, I've had some occasions when I either unplugged the drive before the write-behind buffer was flushed to disc, or accidentally tripped over the usb cable, unplugging it. Being able to plug it back in and not lose any data would have been helpful. In any case, there is a lot to do for this - many, many little details that require you to really understand the disc driver sub-system for Linux thoroughly, including all the kernel-level code that deals with the write-behind cache, drive mount events, etc. As one would find at the edge of a map in the olden days, "There be dragons...".
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •