Attribute system similar to HTTP Headers for local files
Question: I am in the process of writing a program and need some guidance. Essentially,
I am trying to determine if a file has some marker or flag attached to it.
Sort of like the attributes for a HTTP Header.

If such a marker exists, that file will be manipulated in some way (moved to
another directory).

My question is: Where exactly should I be storing this flag/marker? Do files
have a system similar to HTTP Headers? I don't want to access or manipulate
the contents of the file, just some kind of property of the file that can be
edited without corrupting the actual file--and it must be rather universal
among file types as my potential domain of file types is unbound. I have some
experience with Web APIs so I am familiar with HTTP Headers and json. Does any
similar system exist for local files in windows? I am especially interested in
anyone who has professional/industry knowledge of common techniques that
programmers use when trying to store 'meta data' in files in order to access
them later. Or if anyone knows of where to point me, as I am unsure to what I
should be researching.

For the record, I am going to write a program for Windows probably using
Golang or Python. And the files I am going to manipulate will be potentially
all common ones (.docx, .txt, .pdf, etc.)

Thanks in advanced!


Answer: Metadata you wish to add is best kept in a separate file or database for all
files.

Or in another file with same name and different extension or prefix, that you
can make hidden.

Relying on a file system is very tricky and your data will be bound by the
restrictions and capabilities of the file system your file is stored on. And,
you cannot count on your data remaining intact as any application may wish to
change these flags.

And some of those have very specific, clearly defined use, such as creation
time, modification time, access time...

See, if you need only flagging the document, you may wish to use creation
time, which will stay unchanged through out the live of this document (until
is copied) to store your flags. :D

Very dirty business, unprofessional, unreliable and all that.

But it's a solution. Poor one, but exists.

I do not know that FAT32 or NTFS file systems support any extra bits for
flagging except those already used by the OS. Unixes EXT family FS's do
support some extra bits. And even than you should be careful in case some
other important application makes use of them for something.

Mac OS may support some metadata by itself, but I am not 100% sure.

On Windows, you have one more option to associate more data with a file, but I
wouldn't use that as well.

Well, NTFS file system (FAT doesn't support that) has a feature called
streams.

    
    
    In essential, same file can have multiple data streams under itself. I.e. You have more than one file contents under same file node.
    

To be more clear. Same file contains two different files.

When you open the file normally only main stream is visible to the
application. Applications must check whether the other streams are present and
choose the one they want to follow.

So, you may choose to store metadata under the second stream of the file.

But, what if all streams are taken?

Even more, anti-virus programs may prevent you access to the metadata out of
paranoya, or at least ask for a permission. I don't know why MS included that
option, probably for file duplication or something, but bad hackers made use
of the fact that you can store some data, under existing regular file, that
nobody is aware of.

Imagine a virus writing it's copy into another stream of one of programs
already there.

All that is needed for it to start, instead of your old program next time you
run it is a batch script added to task scheduler that flips two streams making
the virus data the main one.

Nasty trick! So when this feature started to be abused, anti-virus software
started restricting files with multiple streams, so it's like this feature
doesn't exist.

If you want to add some metadata using OS's technology, use Windows registry,
but even that is unwise.

What to tell you? Don't add metadata to files, organize a separate file, or
index your data in special files with same name as the file you are refering
to and in same folder.

