GeezerVison is showing github.com — Consider attributes outside of simply Host · Issue #13 · ReconJSON/ReconJSON
Consider attributes outside of simply Host #13DescriptionRecon/OSINT tools usually involve a lot more than just Hosts/IP/Subdomains
I'd consider including:
- Email Address
- Examples:
- Sites found with address
- PGP keys
- URL
- Usernames
- Examples:
- Twitter profiles found
- Github profiles found
- Social media accounts in general
- Associated to email addresses where name is found via PGP key look ups
- Domain names
- File hashes
- ASN
Reacted by Glenn 'devalias' Grant Activitydeadbits commented on Jun 26, 2018 Reacted by Glenn 'devalias' Grant caffix commented on Jun 26, 2018 Rhynorater commented on Jun 29, 2018 Grea thoughts @deadbits We're gonna put some additional thoughts into how to deal with these different objects. Reacted by Adam Swanda and Glenn 'devalias' Grant Rhynorater commented on Jun 30, 2018 @deadbits - Just to follow up with you, I think we are going to use this as a V2 of ReconJSON. Being able to completely describe a Host a fairly big task. Being able to describe everything, is a HUGE task. So we're gonna take it in bite-sized chunks. After Hosts, here are some things on the docket:
- Domains (second-level domain itself, whois data, historic DNS)
- IP ranges
- Users (Usernames, First Name, Last Name, Phone Numbers, Emails, Passwords, Public/Private/Keys (GPG, RSA, DSA, ect...), Generic data)
- External Services (S3 Buckets, DO Spaces, Github, Travis CI, DockerHub, Papaly, ect...)
- AD Forests (This one scares me a lot....)
I've looked into SpiderFoot's data elements, and they are great! Thanks for recommending those. We'll certainly use those in the future.
In regards to File Hashes, I'm not sure we're going to be able to fit that into the scope any time soon. What need to see for storing those and how do you use those in your recon methodology? @deadbits ? Reacted by Adam Swanda and Glenn 'devalias' Grant deadbits commented on Jun 30, 2018 @Rhynorater awesome, thanks for the update and follow up on this. I know it's a pretty big ask overall. I'd be happy to help work with you to figure out ANY of the mappings I recommend whenever you're ready too, just FYI.
For the file hashes, a lot of my personal and professional OSINT/Recon investigations are often based around malware network infrastructures or threat actor specific network infrastructures, and many other people I know do the same.
Take for example a schema used by Maltego where relationships can be made (granted that is a graphDB I believe) but using nesting of objects within a JSON schema properly can provide the same type of relationships without duplication of data
As an example for using file hashes in the context of these other objects, https://shadowdragon.io/blog/malnet-maltego-transforms-with-proofpoint-data/ has a very quick example with screenshots that do the idea more justice than my words :)
Use Cases
Some common use cases here would be the ability to show a connection between :
- A given URL/IP/Host and a file hash downloaded from that location
- A given URL/IP/Host and a file hash that communicates outbound to that location
- A given URL that hosts a file, as represented by the hash
Aka:
- File communicates with host/ip/url
- File downloaded from host/ip/url
- File hosted on URL
Some of the subfields of a Hash object I can imagine being useful per item would be represented
- firstSeen: date object
- lastSeen: date object
- fileNames: list of strings
A basic schema might be something like this
{ "hash": file_md5_or_sha256_hash:
{"communicates_with":
{"http://www.evil.website.command.and.control/gate.php":
{"firstSeen": date,
"lastSeen: date,
"filenames": [filename1, ..]
},
{"downloaded_from":
{"http://www.eviler.website.delivering.malware/payload.exe":
{"firstSeen": date,
"lastSeen: date,
"filenames": [filename1, ..]
},
}
}
}
And so on for each of the other objects that might relate as mentioned above (IP/Host/URL) and (DownloadedFrom, HostedAt, CommunicatesWith). Or you could start from the IP/Host/URL objects as the second level keys and include the aforementioned sub-keys nested below those objects.
If any of that doesn't make sense, which it might not tbh lol I've been low on coffee today, I'll happily hop on my laptop and create a basic schema that could explain this a good bit more. deadbits commented on Jun 30, 2018 Think of services like PassiveTotal that give you a view of Domains, Subdomains, SSL Certs associated with these, and File hashes that relate to the Domains & Subdomains - which I believe is determined by a file being found on a host (url/ip) or communicating with a host (url/ip/domain) deadbits commented on Jun 30, 2018 Just another external resource here but the Recon framework Intrigue is pretty great and supports many entities in this field overall. Here's a link to their Github page of entities / artifacts if you need any more ideas of other use cases and how others integrate artifacts into their schemas
Though Spiderfoot link above has more than enough ideas for a great many number of releases ;-) kcreyts commented on Jul 17, 2018 0xdevalias commented on Jul 21, 2018 I haven't made the time to read all the other threads yet, but noticed this one, and in particular the bit about "fully describing a domain is a big task".
An approach I have taken in a personal project recently is to use a bit of a 'tier'/parent model in my hierarchy.
So there is an apexHost at the root (eg. foo.com), which then has host elements linked to it to handle the actual domain/subdomain part (eg. foo.com, a.foo.com, a.b.foo.com, etc). For each of these there can be more types linked depending on the data I need to store about them, grouped by 'type of data'. For example, maybe one holding information about subdomain takeovers, etc.
A similar model could be applied for IP -> portscan or similar.
Basically ending up with a graph database sort of pattern. But then if you take each 'element' of this graph, you could probably map it back to what makes sense as 'individual' data types for the json side of things. deadbits commented on Jul 21, 2018 @0xdevalias
Definitely a fan of the first described schema. It's a pretty logical way of following host information. Only thing I imagine being a problem is, if I'm following you correctly, is:
- What of the user doesn't start out with the top level domain?
- Apps would have to parse that info automatically on behalf of the user and essentially force them to store it in whatever DB they're using even if they don't want it.
Data duplication, in my example scenario where you don't do the steps above, then becomes a problem too though.
Just thinking out loud... 0xdevalias commented on Aug 19, 2018 @deadbits It depends on the data you're starting with. So ip is it's own 'root'. For a host/subdomain you can always derive the apex. For ports I make an implicit assumption that they're tied to an IP. For an endpoint (eg /foo/bar) I make an assumption it's linked to a host (though potentially it could also link directly to an ip..). Etc.
Basically.. I try to only utilise a 'parent hierarchy' where it makes sense/'has to' be that way. Then there can be cross links between things, eg. a host/ip can have an implicit edge between them (possibly with some form of dns object in between depending how detailed you want to get with it. deadbits commented on Aug 22, 2018 @0xdevalias Few questions here..
-
do you happen to have an example of your current idea / implementation of this schema so far? I think i'd better understand it if I saw it with real data, and I probably wouldn't be asking these questions all the time :)
-
I can see cases where problems might occur depending on what data type you start with, if you are forcing certain data types to always be the apex of a doc. Unless I'm misunderstanding what you meant in your last post, in which case please disregard my ignorance and I pre-retract my below comments :)
- Basically, unless the user starts with an "apex" datapoint, it cannot be the JSON document primary key within an app using this schema, correct? Or the application using the schema would need to automatically derive whatever the apex for that starting datapoint is, create a JSON doc with that as the primary key, and add the original datapoint as a child.
If I'm thinking about this correct, that assumes the apps using this schema would want to perform those actions if a users starts with a non-apex data point AKA I don't think the schema should require an application to make resolutions(or what not) simply because a user starts off with a certain data type.
If you really wanted to re-think this whole approach; handle cross-references and allowing for objects to relate to one another regardless of what was created first or what not - you really might want to look at a graph database schema instead; Where Nodes (domain, IP, email, etc) are connected to one another through Edges (relationships), and the Nodes themselves can have properties as well. In the screenshot below is a relatively poor example I made in ~3 mins of what this may look like for Google.com, it's IP, it's mail server and an alias.. (replace "subdomain" with "subdomain of" in the image.. typo on my part)
0xdevalias commented on Aug 25, 2018
do you happen to have an example of your current idea / implementation of this schema so far
Not really in any 'easily sharable' format, and what I do have probably wouldn't provide any more detail than how I shared above unfortunately.
I can see cases where problems might occur depending on what data type you start with, if you are forcing certain data types to always be the apex of a doc.
...
Basically, unless the user starts with an "apex" datapoint, it cannot be the JSON document primary key within an app using this schema, correct? Or the application using the schema would need to automatically derive whatever the apex for that starting datapoint is, create a JSON doc with that as the primary key, and add the original datapoint as a child.
So basically in my personal implementation, I am only adding/enforcing an apex when it makes sense. So the current implementations off the top of my head:
apex (eg. example.com) -> host (eg. foo.example.com) -> endpoint (eg. /foo)ip (eg. 1.2.3.4) -> port (eg. 80)
Given the apex/host example, if you know a host/subdomain, you are always able to determine it's apex/root, which I sort of think about as 'derivable information'. You don't need to do any resolution/etc for this, it's just sort of inherent in how subdomains work.
For an endpoint, I assume that the information is useless unless I attach it to at least a host/subdomain, at which point I can derive the apex as above.
Similarly, with a port, I assume the information is meaningless without being attached to an ip, and so enforce that hierarchy.
I don't think the schema should require an application to make resolutions(or what not) simply because a user starts off with a certain data type.
If by 'resolutions' you mean needing to talk out to DNS/similar, then I absolutely agree. But I feel like if it can 'inherently resolve' something (eg. get the apex from a domain), then I don't see that as being too big of an issue, so long as it doesn't add 'too much extra work' for the implementer. Eg. In golang it is basically 1 dependency + 1 function call to get the apex.
I'm not 100% sure if having this level of hierarchy/distinction would even necessarily make sense for the spec.. it would depend how we want to layout the data overall. In my case, I sort of chose to do it this way as it makes for an nice database schema and efficient queries.
If you really wanted to re-think this whole approach; handle cross-references and allowing for objects to relate to one another regardless of what was created first or what not - you really might want to look at a graph database schema instead
That is actually exactly the approach I have taken to how I have things laid out in my project. To try and translate my vocab onto your picture above:
Basically an apex is a 'pseudo root' for any 'host' (which might be a root like example.com, or a subdomain like foo.example.com, to tie everything together. ip has a link to host. I don't currently have a concept of alias or similar DNS stuff built in.
When I talk about a host/ip/endpoint/etc, in my app, these are all nodes, that have their own properties/data + edges/relationships. deadbits commented on Aug 25, 2018 Great, that clears a ton of stuff up and makes more sense now. I was originally picturing that your description would be enforcing app functionality (dns resolutions etc) in order to use it / to create an apex if you don't start off with an attribute that is itself an apex - if that makes sense. Your explanation clears it up tho.
Sometimes it's a bit hard to visualize the schema in abstract terms, especially with various connections between attributes - laying it out in a quick image or something helps a lot. 👍 Rhynorater commented on Aug 29, 2018 The apex problem is one that we kind of addressed in #8. It might be interesting to consider an apex attribute in the metadata field.
Otherwise, are there current action items on this thread, if so, could we consolidate them? We already have a framework (via serviceDescriptors) to provide details about services. However, for things like Usernames, it might be helpful to define another top level object like Company that Usernames can be a sub-object of. Unfortunately, I don't think we'll be able to give these other objects much attention, though, until a full V1 of this standard is released, as the goal of V1 is to be able to fully describe Hosts and their properties.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
