[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I'm trying to implement a virtual filesystem with FUSE.
This filesystem consists of many APIs and I have to implement it.
Fortunately, there is sample code at the repository of the library.
So I reference it and write a simple readonly filesystem.
In the second half of the week, I expand it and try to make it writable.
However, I found strange behavior in my program. I have to debug and
write test code.
So that is my objective for next week.

Regards,
Kaoru Esashika

Project Repository: https://github.com/pluser/fusebox

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I implemented some FUSE interface.
Therefore, the emerge program can work well on my Fusebox filesystem.
However, I ran into a few problems when we tried to implement it.
I've written about the problem and explanation about FUSE on my blog.
If you interested, please check
https://blog.pluser.net/en/posts/2020/gsoc-milestone-1/
Now I need to integrate Fusebox to the emerge program. But I don't
familiar with the emerge program well, I need to read the code...

Project Repository: https://github.com/pluser/fusebox

Regords,
Kaoru Esashika

Re: [gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1238 bytes --]

On Sun, 2020-06-21 at 20:17 +0900, Kaoru Esashika wrote:
> Hi,
> This week, I implemented some FUSE interface.
> Therefore, the emerge program can work well on my Fusebox filesystem.
> However, I ran into a few problems when we tried to implement it.
> I've written about the problem and explanation about FUSE on my blog.
> If you interested, please check
> https://blog.pluser.net/en/posts/2020/gsoc-milestone-1/
> Now I need to integrate Fusebox to the emerge program. But I don't
> familiar with the emerge program well, I need to read the code...
> 

The questions I'd like to see answered are:

1. From our talking you indicated that FUSE has both high-level and low-
level API.  However, the post only hints at it.  Could you explain both
shortly and why you've chosen the one you've chosen?

2. How are you planning to deal with the essential problem that
the underlying directory tree may consist of multiple filesystems with
different files having colliding inode numbers?

3. How do other passthrough filesystems deal with the same problem?
 What are the advantages/disadvantages of different approaches you've
considered, and why did you chose this solution?


-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Thank you for the reaction, Luca, Michał.

I will expand my blog post soon. Thank you, Luca.

I want to answer the question from Michał.

> 1. From our talking you indicated that FUSE has both high-level and low-
> level API.  However, the post only hints at it.  Could you explain both
> shortly and why you've chosen the one you've chosen?

There are some FUSE libraries which I can choose. But actively
maintenanced ones are pyfuse3 and libfuse (both are developed by the
same person).
pyfuse3 is an asynchronous python library. I call it 'low-level'
because functions that should be implemented are called with inode
arguments.
libfuse is a C library. It has two interfaces, high-level synchronous
one and low-level asynchronous one. The difference between the two is
whether arguments are path or inode.
So, with the exception of libraries that are no longer maintained, the
only choice is pyfuse3.

> 2. How are you planning to deal with the essential problem that
> the underlying directory tree may consist of multiple filesystems with
> different files having colliding inode numbers?

VFS in the kernel and programs in userland expects consistency with
regard to inode, so Fusebox should provide a unique inode for each
file.
Fusebox can do this by finding out which device the file belongs to.
Then remember mappings from/to device-inode pairs to/from 'virtual'
inode.
However, I worried this solution hits performance since it potentially
causes many stat() system calls.

> 3. How do other passthrough filesystems deal with the same problem?
>  What are the advantages/disadvantages of different approaches you've
> considered, and why did you chose this solution?

I investigated the high-level interface of libfuse. It caches the
reply of the user program. and automatically convert inode and path.
So the high-level interface of libfuse will affect the problem.
I also investigate sandboxfs ( https://github.com/bazelbuild/sandboxfs
). I think it take same strategy to Fusebox.
sandboxfs construct filesystem tree virtually on memory and notice
virtual inode to user application.
This approach may hit performance because of frequent stat() call, but
I cannot think better approaches.

Thank you for your reply and support.

Would you mind if I publish this mail on my blog, Michał? It was a
worthwhile discussion and I'd like to share it on my blog.

Regards,
Kaoru Esashika

Re: [gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2940 bytes --]

On Tue, 2020-06-23 at 01:17 +0900, Kaoru Esashika wrote:
> Thank you for the reaction, Luca, Michał.
> 
> I will expand my blog post soon. Thank you, Luca.
> 
> I want to answer the question from Michał.
> 
> > 1. From our talking you indicated that FUSE has both high-level and low-
> > level API.  However, the post only hints at it.  Could you explain both
> > shortly and why you've chosen the one you've chosen?
> 
> There are some FUSE libraries which I can choose. But actively
> maintenanced ones are pyfuse3 and libfuse (both are developed by the
> same person).
> pyfuse3 is an asynchronous python library. I call it 'low-level'
> because functions that should be implemented are called with inode
> arguments.
> libfuse is a C library. It has two interfaces, high-level synchronous
> one and low-level asynchronous one. The difference between the two is
> whether arguments are path or inode.
> So, with the exception of libraries that are no longer maintained, the
> only choice is pyfuse3.
> 
> > 2. How are you planning to deal with the essential problem that
> > the underlying directory tree may consist of multiple filesystems with
> > different files having colliding inode numbers?
> 
> VFS in the kernel and programs in userland expects consistency with
> regard to inode, so Fusebox should provide a unique inode for each
> file.
> Fusebox can do this by finding out which device the file belongs to.
> Then remember mappings from/to device-inode pairs to/from 'virtual'
> inode.
> However, I worried this solution hits performance since it potentially
> causes many stat() system calls.

This sounds a bit like premature optimization problem.  Let's focus
on getting a working solution first.  You can look into improving
performance with different inode solution later on.

> 
> > 3. How do other passthrough filesystems deal with the same problem?
> >  What are the advantages/disadvantages of different approaches you've
> > considered, and why did you chose this solution?
> 
> I investigated the high-level interface of libfuse. It caches the
> reply of the user program. and automatically convert inode and path.
> So the high-level interface of libfuse will affect the problem.
> I also investigate sandboxfs ( https://github.com/bazelbuild/sandboxfs
> ). I think it take same strategy to Fusebox.
> sandboxfs construct filesystem tree virtually on memory and notice
> virtual inode to user application.
> This approach may hit performance because of frequent stat() call, but
> I cannot think better approaches.
> 
> Thank you for your reply and support.
> 
> Would you mind if I publish this mail on my blog, Michał? It was a
> worthwhile discussion and I'd like to share it on my blog.
> 

Sure, please include anything that can help your readers understand
what's going on with the project and why.


-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I'm struggling with bugs.
For example, Internal inode-path mapping is incorrect when link()
function is called after forget() was called.
Because such a phenomenon is very rare, difficult to reproduce.
Anyway, I fixed some bugs and I have successfully built GNU Hello[1]
on the Fusebox.
GNU Hello is a minimum, simple 'hello world' project which is
maintained for practice.
Because it is minimum and contains standard build process
("/.configure; make; make install"), I choose to initial target.
In the next week, I will write unit tests to keep working the code.
and write code to verbose output for evaluation.
Thank you.

[1]: https://en.wikipedia.org/wiki/GNU_Hello
Fusebox Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I'm trying to fix some issue which is pointed out by my
mentor (thanks!)
Some of the issues are caused by deadlock. Since Fusebox is a single
thread program,
It cannot process a request from VFS (kernel) while processing another request.
So I choose a solution that hides the directory which causes deadlock.
Because Fusebox has been developed for sandbox, basically it doesn't
need recursive access.
So I think this not bad.

In the next week, I will write unit test because debugging and testing
burden me.
And also I will implement virtual inode mechanism (which will kick out
a compromise in the code)

Project Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I'm coding about vnode related code. 'vnode' is virtual inode.
Passing through the base file system inodes as is may cause inode conflicts.
So I have to 'convert' inode to vnode. And I was writing that code.
I also refactored the code because I feel code quality is not good.

Next week, I will write the code about unit test and ACL (Access
Control List) feature.

Project Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I was writing unit test code.
The test code was harder to write than I expected.
In particular, the code that was associated with the filesystem had to
be patched with system calls to the OS and replaced with mock objects.
Python has a built-in mock object, and all calls to this object can be
made in You can record it. This feature is very useful and we use it a
lot in our test code.

I'd like to have even higher test coverage, but I can't just write
test code, so next week I'm going to implement the ACL (Access Control
List) feature.

Project Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I wrote the code about ACL (Access Control List).
The ACL allows you to actually control whether or not the application
can access your files.
This implementation also includes an interface that allows you to
control access to the files dynamically.
Specifically, you can control access to specific files by writing a
list of files to be controlled in a special virtual file called a
control file.

Next week, I will integrate the Fusebox with emerge/portage. And also,
I need prepare to evaluation...

Project Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika

Re: [gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[EBo]

Recently I was watching some videos that was looking at vulnerabilities 
in IP camera systems and many of the fails that the security person was 
able to exploit were forgetting to lock down access to some directory or 
file so that he was able to first examine a program or script, and then 
determine points of access.  With the discussion here I was wondering if 
there was any mechanism to turn all access off, and then 'grant' access 
to something.  This may be similar to how Gentoo's USE flags can be 
likewise cleaned by:  "USE = "-* X alsa..."

Anyway, I browsed your tests and did not find anything and thought I 
might mention it.

   EBo --

On Jul 26 2020 5:00 PM, Kaoru Esashika wrote:
> Hi,
> This week, I wrote the code about ACL (Access Control List).
> The ACL allows you to actually control whether or not the application
> can access your files.
> This implementation also includes an interface that allows you to
> control access to the files dynamically.
> Specifically, you can control access to specific files by writing a
> list of files to be controlled in a special virtual file called a
> control file.
>
> Next week, I will integrate the Fusebox with emerge/portage. And 
> also,
> I need prepare to evaluation...
>
> Project Repository: https://github.com/pluser/fusebox
>
> Regards,
> Kaoru Esashika

Re: [gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi, Thank you for your advice and for reviewing my code, EBo.

I agree with your advice, the whitelist method is good practice.
In my current code, the default security model is the whitelist method.
However, in the test code, I change it to the blacklist method for convenience.
So I guess you might misunderstand.

Because the behavior of ACL should be maintained to Gentoo's sandbox,
this behavior might be changed for the future.
Anyway, thank you for your advice. I'll keep your advice in mind.

On Mon, Jul 27, 2020 at 1:48 PM EBo <ebo@sandien.com> wrote:
>
> Recently I was watching some videos that was looking at vulnerabilities
> in IP camera systems and many of the fails that the security person was
> able to exploit were forgetting to lock down access to some directory or
> file so that he was able to first examine a program or script, and then
> determine points of access.  With the discussion here I was wondering if
> there was any mechanism to turn all access off, and then 'grant' access
> to something.  This may be similar to how Gentoo's USE flags can be
> likewise cleaned by:  "USE = "-* X alsa..."
>
> Anyway, I browsed your tests and did not find anything and thought I
> might mention it.
>
>    EBo --
>
> On Jul 26 2020 5:00 PM, Kaoru Esashika wrote:
> > Hi,
> > This week, I wrote the code about ACL (Access Control List).
> > The ACL allows you to actually control whether or not the application
> > can access your files.
> > This implementation also includes an interface that allows you to
> > control access to the files dynamically.
> > Specifically, you can control access to specific files by writing a
> > list of files to be controlled in a special virtual file called a
> > control file.
> >
> > Next week, I will integrate the Fusebox with emerge/portage. And
> > also,
> > I need prepare to evaluation...
> >
> > Project Repository: https://github.com/pluser/fusebox
> >
> > Regards,
> > Kaoru Esashika
>
>

Re: [gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[EBo]

You are welcome.  I understand the utility and usefulness of both he 
white and black lists.  Maybe you can add 2 tests which tests that the 
white/black listing respect the boundary.  It will also show that you 
can go either way, and them maybe in the docs suggest which might be 
preferred in what circumstance.

HHHhhhh... As a further note.  Many projects and groups are moving away 
from what some consider racially charged terminology -- and if I am not 
mistaken white/black lists may be one of them.  Can you check and make 
sure what the current acceptable terminology is and we all might want to 
get into the habit using them.

On Jul 28 2020 2:08 PM, Kaoru Esashika wrote:
> Hi, Thank you for your advice and for reviewing my code, EBo.
>
> I agree with your advice, the whitelist method is good practice.
> In my current code, the default security model is the whitelist 
> method.
> However, in the test code, I change it to the blacklist method for 
> convenience.
> So I guess you might misunderstand.
>
> Because the behavior of ACL should be maintained to Gentoo's sandbox,
> this behavior might be changed for the future.
> Anyway, thank you for your advice. I'll keep your advice in mind.
>
> On Mon, Jul 27, 2020 at 1:48 PM EBo <ebo@sandien.com> wrote:
>>
>> Recently I was watching some videos that was looking at 
>> vulnerabilities
>> in IP camera systems and many of the fails that the security person 
>> was
>> able to exploit were forgetting to lock down access to some 
>> directory or
>> file so that he was able to first examine a program or script, and 
>> then
>> determine points of access.  With the discussion here I was 
>> wondering if
>> there was any mechanism to turn all access off, and then 'grant' 
>> access
>> to something.  This may be similar to how Gentoo's USE flags can be
>> likewise cleaned by:  "USE = "-* X alsa..."
>>
>> Anyway, I browsed your tests and did not find anything and thought I
>> might mention it.
>>
>>    EBo --
>>
>> On Jul 26 2020 5:00 PM, Kaoru Esashika wrote:
>> > Hi,
>> > This week, I wrote the code about ACL (Access Control List).
>> > The ACL allows you to actually control whether or not the 
>> application
>> > can access your files.
>> > This implementation also includes an interface that allows you to
>> > control access to the files dynamically.
>> > Specifically, you can control access to specific files by writing 
>> a
>> > list of files to be controlled in a special virtual file called a
>> > control file.
>> >
>> > Next week, I will integrate the Fusebox with emerge/portage. And
>> > also,
>> > I need prepare to evaluation...
>> >
>> > Project Repository: https://github.com/pluser/fusebox
>> >
>> > Regards,
>> > Kaoru Esashika
>>
>>

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I brushed up ACL (Access Control List) feature.
I implement a compatibility interface to keep the consistency of the
current portage's sandbox.
Also, I wrote many tests and stabilized the feature.
The unit tests make me to easier modifying code. Because this is to
reduce the need to check regressions as the code changes.

Next week, I will write some code to integrate with portage.
It's going to be a different kind of challenge, but I'm going to do my best.
And I have some ideas about ACL interface. I'd like to implement it if
I had the time.

Project Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I implement the interface to control ACL.
This interface is file-oriented.
I thought it would be better to be able to open the file in an editor
and edit the list directly, so I implemented it.
Changes are passed directly to Fusebox and take effect immediately.

Next week, I will integrate with the portage.
Integrating may require patching to emerge program, so I might
encounter some problems.
I'm going to work on this as soon as possible for the final evaluation.

Project Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I'm writing glue code to integrate with emerge.
Now I can call the Fusebox from emerge process and working with ebuild.sh.
However, there is a problem that emerge process access a file that is
unlinked already.
I have to investigate this phenomenon and it may spend a lot of time...

Anyway, next week, I will investigate and debug Fusebox.

Project Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika

[gentoo-soc] Weekly Report: Fusebox - FUSE Porwered sandbox project

[Kaoru Esashika]

Hi,
This week, I'm fixing many bugs to integrate with emerge.
I realized bugs by integrating with emerge.
I've done a lot of research on pipes and standard inputs and outputs.
Making multiple processes work together is harder than I thought it
would be, but I'm going to do my best.

Project Repository: https://github.com/pluser/fusebox

Regards,
Kaoru Esashika