Practical activity

File systems and permissions

Introduction

A few points:

• A file is a (possibly empty) set of bytes, associated with a name and possibly metadata (such as date, owner, etc.)
• A directory is a (possibly empty) set of files or subdirectories. It is itself associated with a name and metadata. Files can then be arranged hierarchically in directories and subdirectories.
• Operating systems generally adopt certain strategies in the way they choose to organize their file hierarchy. (Where to store system resources, configuration files, applications, libraries, user data, etc.)
• Files are recorded on a storage device, such as a hard disk, a solid-state drive (SSD), a USB key, a DVD, a plain old floppy disk, a magnetic tape, etc. (Depending on the underlying technology, a storage device can be divided into parts called partitions or volumes.)
• Some of those storage devices are local (physically inside or plugged on your computer), or on a distant computer and usable via the network.
• The way bytes are organized on the storage is called a file management system (decides which bytes are for the name, the data, the metadata, etc.)
• Some operating systems extend the concept of files in several directions:
  • links or shortcut: the possibility for a file to have several names. When certain nodes of a hierarchy can be reached by several paths, we obtain a graph. This extends the way files can be organized.
  • special files: Linux, Mac and Unix in general like to use file names as entry points to kernel services. These file names are not associated with stored data, but are associated to specific attributes or metadata. So that, when an application performs read-write operations, data is transmitted to the corresponding device driver in the kernel, which then activate the device. In other cases, this enables the use of socket services, pipelines, shared memory and so on. In yet other cases, this allows you to dynamically configure or inspect the kernel. Well, this is special. For now, it’s enough to remember that this exists.

Explore the docs

So, according to the documentation and Wikipedia pages mentioned above, what should be the name of the file system used on your own computer? (FAT32, NTFS, Ext4, APFS, etc.)

Explore your drive

It’s your own computer. Look at an application named “File Explorer”, “File Manager”, “Files”, “My Computer”, “Finder” or similar. (Typically the icon looks like a papers folder or a filing cabinet.) Know how to launch this application. Keep in mind that this is just an application. (You may install others that do the same.) It gives you a nice view of the files stored on your computer; it uses the file system services provided by the operating system, but this app is not the operating system.

So, this application lets you navigate your computer’s file hierarchy. Please, launch it.

• First check your current position, how to view this information.
• Then try navigating up, down, sideways. Think of a tree, with branches, roots and leaves.
• Try to return to your very first location.
• You can also take a look at this application’s configuration options, which may give you some ideas. Just be curious.

On your journey through your files, you’ve come across various categories: system resources, applications, libraries and user data. Did you find out who’s who?

Be comfortable with your file hierarchy

Try command-line

The command line isn’t reserved for movie scenes with geek stereotypes, it’s actually a practical way of using your computer.

In a nutshell, a command line interface (CLI) is an application (or part of an application) offering text-based interaction between human and machine. The human is prompted to type a sentence, the machine understands it (or not) and provides its answer. For this to work, the sentence must be simple and strict, essentially a command word followed by a few parameters or options. The human must know the words that the machine can understand, and type them with his fingers on the keyboard without clicking the mouse. (This extraordinary skill is probably at the root of these cinematic stereotypes.)

In the past, the command line interface was a physical console, named a terminal, a device consisting of a cathodic screen and a keyboard, connected to a cabinet-sized computer. (A great commercial success of the Teletype Corp. in the 70s, which left the common word TTY for these devices.) Now, it’s just an app on your PC. Thus, “command line”, “cli”, “shell”, “console”, “rollover”, “terminal”, “tty” are more or less synonyms with minor peculiarities.

Let’s try.

Permissions

Your computer is certainly a multi-user system. This means that several users can use it (simultaneously or alternately), and that the system does its best to preserve the integrity of everyone’s data. Users here can be a physical person, or not: on your computer you play two roles, the role of the system administrator (install, uninstall, configure the system and apps), and the role of a simple user (happy eyball user which click and enjoy, and code). Modern systems therefore implement ways of isolating and separating things. First at the login: before starting your session, the computer asks you who you are and to prove it (e.g. username + password). Then, during your session, the system prevents you from damaging other people’s files (accidentally or deliberately), unless explicitly authorized. Also, it’s generally considered a bad idea to open its session as administrator, or to play user applications as administrator. (Even if you are the administrator of your computer, create a normal user account and use it to log in for normal use. Limit your role as administrator to a very limited time.)

Generally speaking, the operating system takes care of this through its file management system; each file is stored with metadata indicating in one way or another which user is authorized to read, write or execute the file. The details depend on the system in question.

Note also that permissions on directories must be understood with care.
Being authorized to read a directory means being authorized to obtain a list of the file and subdirectory names it contains.
Being authorized to write to a directory means being authorized to add or delete a file in that directory, independently of being authorized to access the file’s contents.
Being authorized to execute a directory is nonsense (unlike a file, which can be an application). It is replaced by the idea of being authorized to traverse it, i.e. to access its content without the authorization to read it.

$ ls -l
total 8
-rwx--x--x   1   clohr   info-perm     92   juil. 15 17:33   helloworld.py*
-rw-r--r--   1   clohr   info-perm     33   juil. 15 17:34   readme.txt
drwx------   2   clohr   info-perm   4096   juil. 15 17:35   session1/

$ echo "Hello world" > dummy.txt
$ ls -l ./dummy.txt
-rw-rw-r--   1   clohr   info-perm   12   16 juil. 09:22   ./dummy.txt

$ chmod go-r ./dummy.txt
$ ls -l ./dummy.txt
-rw--w----   1   clohr   info-perm   12   16 juil. 09:22   ./dummy.txt

$ chmod a+x ./dummy.txt
$ ls -l ./dummy.txt
-rwx-wx--x   1   clohr   info-perm   12   16 juil. 09:22   ./dummy.txt

...

$ cd
$ ls -la
...

Know where to find downloaded files and various application files.

Applications use and produce files, but where are they?

Consider a first scenario.

You visit a web page. You’ve read the documentation carefully. You’re now convinced that this is the thing you do need. You click on download.

What append?

Your web browser (which is just another application) downloads the requested file and stores it somewhere on your computer. Where has this new file been placed in your file hierarchy?

• Open your File Manager. On the displayed file space you get something called “Downloads” (or similar). This is probably where your web browser saved the downloaded file. (Check.) But is this really the right answer to the question? Where is this “Downloads” subdirectory in your file hierarchy? In fact, you see it into a folder named “Homes” (or similar, like a shortcut, to “improve the user experience”). But a shortcut to where? How or what made this shortcut?
  Your File Browser certainly also propose an item called “This PC” or “System” (or similar), an entry point into your computer’s file hierarchy, offering a concrete view of files and their location (without hiding strategies to enhance the user experience). So, browse it to find your downloaded file. Got it? Keep in mint the true location. Please also note that the above-mentioned shortcut is just a trick performed by your file manager, for display purposes.

• Open your Command Terminal (or the name in use on your system). Repeat the exploration, but using the command line interface. Use the commands described above to browse directories, and find the downloaded file. Keep in mind the current location.

• Security considerations.
  The items you have downloaded may content executable. If you click on it, you ask your computer to follow instructions decided (coded) by someone else. (Okay, so everything else on your computer already is.) But, are you sure that this executable is good for you? That there is no hidden action? Your system may (or may not) incorporate an “antivirus”, i.e. an application that examines executable instructions and detects (or not) inappropriate code. It increases trust, but it’s not magic. The first line of protection is you. Have you read the documentation carefully? Did you download it from a reliable source? Is it the very original source and not a third part site (which could inject some extra code in it)? Keep these questions in mind when you click ok. There is probably no definitive answer, just some ways to elevate trust.

Consider a second scenario.

Now you’re starting to become a coder (and soon a developer), with Python, Java, etc. and an IDE (eg. VS Code). So, you played with an IDE. This is an application, which store files. But where?

Same questions as for the first scenario:

• Use your file manager to find the actual location of your code files in the file hierarchy.
• Use your command line interface to find the actual location of your code files in the file hierarchy.

Organize your own user space by lessons/sessions/etc.

You probably don’t put your pants, socks, bathing suits and coats in the same drawer. You’re probably trying to organize your belongings to bring together items that go together.

Please do the same with your digital belongings.

Your session may already propose an organization: “My Music”, “My Videos”, “My Pictures”, “My Downloads”, “My Desktop”, “My Documents”, … This is an organization per type of files. (Well, Documents is rather fuzzy, and there is no “My Codes”.) You may therefore prefer another strategy for organizing the files you’ll have to manage during your training. Why not organizing this by subdirectories corresponding to disciplines / lessons / sessions / activities. It’s certainly not a good idea to have all your working files in the same folder, in a flat way, and to have lots of things simply named “exercise” “project” “report” (of what??). Try to find your own strategy for organizing your files and follow it.

Do it now. You’ve already created files in previous sessions. So create subdirectories, now, with names that make sense to you, and move files into them. Don’t worry if you change your mind and opt for another strategy latter, you can always reorganize and move your files accordingly. Just be consistent.

Manage archives

An archive is a big file containing several files. It is useful to manage backups, or to distribute a project.

In addition to file collection, some solutions also offer compression. This can be done separately or in combination. Password protection can also be offered. It is traditional in the Unix world to use the “tar” format (Tape ARchiver, historically used for tape backups), followed by compression (gzip, bzip2, xz, …) One get file extensions such as “.tar.gz” “.tgz” “.tar.bz2” “.tar.xz” etc. It is traditional in the Windows world to use the “.zip” or “.7z” format (or other proprietary formats such as “.arj” or “.rar”). But many format exist, with differences in terms of access latency, compression ratio and licensing. Most of the different formats are available on all platforms.

The basic operations are: create, list, extract. Most also allow you to add a file to an existing archive, replace a file with a new content or extract one file without extracting the whole.

1. Opening a file archive.

   This is typically the operation you perform to obtain the files provided by teachers for a practical course.

   Download archive_tuto.zip or archive_tuto.tar.bz2 proposed here (or both), and save it in your workspace. If your system offers to open it directly with an Archive Manager (or similar), refuse and choose to save it. Now, you know where your downloaded files are stored and how to access them with your file manager. So, go there and click on it. If your system doesn’t offer an Archive Manager, you can install 7-zip.

   The Archive Manager allows you to list the content of this archive. Note that there are two schools of thought when it comes to building archives: should all content be placed in a subdirectory within the archive, or should it be placed directly at the archive root? This will have a practical impact at the time of extraction. In the first case, all extracted content will be stored and grouped together in an appropriate subdirectory. Whereas in the second case, the files will be mixed with those already existing in the working directory. In this case, it is advisable to prepare by creating a subdirectory for archive extraction. So, the first thing to do is to look at the list of archive contents and check its internal organization.

   You can now proceed with file extraction. Decide where to extract the files, and remember it.

   Unix users may prefer command line tools: tar and unzip are your friends. (Read manual pages! to list: “tar tvf <archive.tar.bz2>” “unzip -l <archive.zip>”; to extract: “tar xvf <archive.tar.bz2>” “unzip <archive.zip>”) Then, look at the new files.

2. Creating a file archive.

   This is typically the operation you perform when you hand in your work to the teachers.

   Create a subdirectory (eg. “my_homework”). Create a few files in it (eg. “foo.py” “bar.txt” …) Return to the previous directory, i.e. the one containing “my_homework”. Your favorite archive manager can be integrated into your file manager. In this case, when clicking on an item (eg. “my_homework”), it proposes a new action such as “archive”. Otherwise, you have to launch it from your applications’ menu. Basically, these archive managers propose to select files or directories on your disk, then ask you for an archive name and a location to create the archive.

   Unix users may prefer command line tools: tar and zip are your friends. (Read manual pages! to archive: “tar cvfj <archive.tar.bz2> <my_homework/>” “zip -r <archive.zip> <my_homework/>”)

   First, take a look at the contents of the archive you’ve created, at the list, to make sure you haven’t forgotten to place any files there. Then, send your archive file to your classmate and ask him or her to take a look at it.

File versioning

File versioning consist in keeping different iterations of the same file. We sometimes do this by hand, making copies of the same file and adding a version number or date to the name of the copies. So, one can go back to an earlier version if necessary, or trace the history of modifications to understand how things got to this point. This can be crucial when working on something that takes a long time to get to a perfect state (e.g. reports or software code). It can also help several people to work on the same file, managing different versions for each contributor. Well, file versioning management is an interesting feature. It’s even better to have a tool for this, and not to do it by hand.

There are many tools, but we’ll concentrate on “git”, which is nowadays widely used by developers.

1. Install
   Follow the official instructions corresponding to your computer. The default configuration options are fine for us.

2. Local use
   Open your favorite command line interface and tell “git” who you are (this will appear in the contributor log when editing files).

   $ git config --global user.name "Your Name"
   $ git config --global user.email "you@imt-atlantique.net"

   Please replace with your own real name and email, don’t blindly copy and paste…
   
   Now create a directory for a dummy project, initialize it for git (this creates hidden configuration files), produce files there, ask git to manage them, then produce a first “commit” (a version), and check that everything is okay at each step.

   $ mkdir git_trial_1
   $ cd git_trial_1
   $ git init
   $ echo "Hello" > readme.txt
   $ echo "print('hello world')" > hello.py
   $ dir
   $ git status
   $ git add *
   $ git status
   $ git commit -m "My first steps with git"
   $ git status

   Then, make a change and produce a new version.

   $ echo "world" >> readme.txt
   $ git status
   $ git commit -a -m "Improved documentation"
   $ git status
   $ git log --stat

   The “git log” command displays the different versions of your project. Note that commits are identified by a random unique name, not a simple number. (Making contributions to a project is not something strictly linear, but rather resembles a task graph. This is why using numbers would make no sense.) Note that the very last commit has for nickname “HEAD” (“HEAD~” is the commit before, and “HEAD~~” or “HEAD~2” as the commit even earlier, and so forth).
   To know what happened to a file:

   $ git blame readme.txt

   Let’s say you have doubts about the changes on this file and want to see a previous version:

   $ git show HEAD~ readme.txt

   It’s probably not that obvious at first glance when looking at the result, but in reality git does all this by applying diffs. The core of the process is calculating and applying diffs on files.

3. GitLab

   You might be interested in pushing your work to a server. (To work with others, or with yourself but from different computers. The server acts as a kind of synchronization point.) You may have already heard of GitHub. Note that git is not GitHub. GitHub is a server for git projects from the company GitHub Inc., acquired by Microsoft in 2018. But other git servers are possible. It is important to know who you are pushing your work towards.

   IMT Atlantique offers a GitLab server for educational purposes. Log in and create a project, with a name like “git_trial_2”. You can use a predefined template, e.g. “Sample GitLab Project” Initialization may take a minute or two. Be patient. The server, in addition to git, offers several features such as a wiki, a web space, tickets, etc.

   In the upper right corner of your project page, there is a blue button named “Code”. Expand the associated menu. Several ways to clone/open/download your project are offered. Let’s choose “Clone with HTTPS”. Keep the URL provided in your clipboard.

   Open your favorite command line interface and clone this GitLab project to your computer.

   $ git clone https://gitlab-df.imt-atlantique.fr/xxxx

   Use the URL provided from your clipboard (and note the above ‘xxxx’).

   You can now go to this project directory, make any changes you need, locally on your computer, do commits and other fancy git tasks, and occasionally push that to the server. The corresponding command is “git push”, and its friend “git pull”. You can do this from multiple computers. (Note that we do not intend to detail here how to resolve complex merge conflicts. But if you’re working on separate files, on separate parts of files, or if you’re following the strict pull-commit-push sequence, there probably won’t be a problem.)

   $ cd git_trial_2
   $ git pull
   $ echo "foo bar" >> doc.txt
   $ git add doc.txt
   $ git commit -m "add documentation"
   $ git push

   Note that the push and pull commands work because this git project was configured with a URL.

   $ git remote get-url origin

   
   

   Your “git_trial_1” has no remote URL. Let’s associate it with a GitLab project. Connect to the GitLab server. Create another project, with a name like “git_trial_1”. But this time it is very important to chose “Create a blank project”.
   Then, GitLab provides some guidance on how to connect your local project to GitLab’s. Open your favorite command line interface and follow them.

   $ cd git_trial_1
   $ git remote add origin https://gitlab-df.imt-atlantique.fr/yyyy
   $ git branch -M main
   $ git push -uf origin main

   You can now go to this project directory, make any changes you need, locally, do commits and other fancy git tasks, and occasionally push that to the server…

These first steps with git are a very minimalist introduction. This is sufficient for a single-user context, for small projects, without complex file editing conflicts. But this may cover most of our needs for now. More documentation from the official tutorial or from third parties. You could learn how to create branches, tags, merges and many other fabulous things.