Description

jobertabma

opened on Jun 26, 2018

Issue body actions

Hi team,

Thanks for putting this together! I wanted to give some high level feedback on the current implementation and I have some questions about its future and wanted to share some of my thinking on it. Feel free to ignore things you disagree with or don't believe are important. Some of it might've come up in other issues already. I scanned them briefly and will leave out things that others have already brought up. I applaud the effort and would love to be involved in some way!

• What do you think the general application for this will be? How will applications consume this information and how will you gain traction / adoption? My thinking is that this will spark interests for people to write converters and mergers and optimize for their own use cases, but I'm failing to see the use cases of it right now (I have some thoughts though, see below).
• There seems to be an opportunity to drive adoption by enabling organizations to have them provide the recon information in this format, rather than doing it the other way around. What are your thoughts on this? E.g. what if an organization could provide this information to participants of their bug bounty programs or ahead of a penetration test?
• With the current scope, this is a really big project. Are you planning to keep it small and, if so, what would the use case be? What would a first stable version look like? E.g. making this generic enough for broad adoption is extremely difficult and will be a lot of work for its maintainers.
• I believe that one of the bigger benefits of having a schema definition for this kind of information is that it allows itself to be checked into version control. This will help with providing proof, determining changes over time, and working together with others. Collaboration isn't outlined in the project yet, but I feel that if this were to see adoption, I think that would become key to its success.
• When you write a specification like this, I'd recommend to do everything to avoid ambiguity. Reading the spec right now leaves a lot of room for people their own interpretation. When this is getting to a stable state, I'd spend more time on that (e.g. a path is a valid URI, but so are #hello_world and https://a:b@c:12).
• Right now you're focussed on layer 3 and layer 5 information of a host. Have you thought about splitting this up and make it easier to distinguish layers from each other? Right now the specification seems focussed on bug bounty, whereas, for example, layer 2 information might be useful when performing tests inside networks. This may also increase adoption because it becomes more useful during penetration tests and allow it to become an actual specification instead of just a schema definition.
• Given that the data that'll be stored using the specification is volatile, I'd think it'd be useful to have a generic set of attributes that give some insight into when the information was discovered, like last_observed_at which would contain an ISO 8601 date/time notation.
• It's hard to reason about some of the current design decisions so far. For example, using camel casing and instead snake casing for attributes like a set of hosts would avoid problems with language constraints, e.g. where camel cased strings are referred to or interpreted as constants, like in JavaScript. Related to that, using singular words for arrays is slightly confusing (hosts > Host). Having this around will help you justify certain design decisions later on as the project matures.

I think there are a number of rabbit holes here that you may want to avoid, and they're mainly around scoping and getting this to a first stable version. I'd encourage you to define the use cases and scope first and then write the spec. Right now it's unclear and feels too detailed / prone to be nitpicked upon.

Feel free to close this issue and merge it into separate threads if that's easier. I also jotted down some thoughts on the (technical) implementation, but I'd love your feedback on this first so we focus on the right things given that it's so early on in the project.

Activity

1. Rhynorater commented on Jun 26, 2018

   

   Rhynorater

   on Jun 26, 2018

   Member

   More actions

   @jobertabma - this is awesome feedback. Thanks so much! Here are my thoughts point by point.

   • The idea for this is a universal format that can be fluidly be imported and exported from different tools. For example, It would be great to run a set of subdomain enumeration tools like so: Subfinder > output.json -> Amass -> output.json -> MassdnsBruteForcer -> output.json -> MassScan -> output.json -> gobuster -> output.json. Each different tool is making a modification to output.json and adding whatever it can add. The key piece of this being that the standard defines a seamless input and export from each tool. With respect to adaptation, we've already got a bunch of big tool developers on-board (subfinder, amass, aquatone, ect...) the hope is that the rest will hop on once the format gains inter-functionality. If not, it would be feasible to create a command-line utility to go along with the format to convert non-conforming tool's output into ReconJSON format.
   • Yeah - that would be phenomenal. The hope is that we can get this adopted at an industry level.
   • I believe the answer to this question is both yes and no. While I don't plan to keep this standard particularly small (I'd love to have multiple contributors/admins and gain large adaptation), I would like to keep it rather simple. IMHO I don't know that there is a ton to recon data - you've got ips, dns, ports, and then the rest is application protocol level - which I think we might shy away from beside basic descriptors such as banners and protocol name (I may have gone too far with content-brute forcing. Thoughts?)
     The first stable version would look like a standard that is very specific (perhaps even regexes defined for each field) that is able to describe the following aspects of a host: ip, dns configuration, ports open, timestamp for data, methods used to aquire this data. That's my thought.
   • One of my top priorities right now is to get a collaborative environment set up for the standard and get some other people invested enough to help me maintain this. While I may have been the one to set the wheel rolling, and while I am certainly invested in the success of the standard, there are certainly more qualified and experienced people in creating and maintaining a standard like this * cough * Ed * cough *. I'd love any suggestions on how to do this better and I'm more than happy to hand over contributor rights. My only concern here is I don't want it to become chaotic and cause a mess. Definitely dig the source control and delta ideas.
   • I absolutely agree. It's been about a day since this has blown up, and I've spent just about every minute trying to read on how to do this sort of thing :P I'm thinking that I will defined each field using a set of regular expressions guided by the RFCs for the protocol attached to each field. For example, DNS should be 253 characters, but not more than 63 per partition seperated by a dot. I am learning about RFC "MUST/SHOULD/ETC" format as well for specifity purposes.
   • Interesting thought. I think it would be reasonable to go down to the MAC address level (this could be helpful for identifying different system's manfacturers) but I can't think of anything else in layer 2 that would be useful. Any ideas?
   • I totally agree. As @michenriksen mentioned, we should certainly have a timestamp in the standard. This timestamp should be on each object. Up for debate, however, is if historical data (dates of modification) should be added to this timestamp as Michael suggested. This would bloat up the datastructure a good bit, but would also provide some very interesting insights.
   • For sure. I think there should be some continuity added. I kinda just stuck a Minimum Viable Product out there because the project was getting some momentum and I wanted to get community input (like this). There is certainly some design factors that need to be well thought out and standardized

   Thanks again for contributing, and I'd love to hear your further thoughts on the technical implementation.

   Reacted by Glenn 'devalias' Grant
2. Rhynorater

   self-assigned this

   on Jun 26, 2018
3. deadbits commented on Jun 30, 2018

   

   deadbits

   on Jun 30, 2018

   More actions

   "Given that the data that'll be stored using the specification is volatile, I'd think it'd be useful to have a generic set of attributes that give some insight into when the information was discovered, like last_observed_at which would contain an ISO 8601 date/time notation."

   I highly agree with this. Including fields such as last_observed / first_observed aka first_seen and last_seen would be a huge help. Many existing platforms include this already using one field name or another but the results is the same.

   Depending on the artifact, you could use a submission_dates field to identify when/if this artifact was seen more than once and users could simply count that list to get a unique count on submissions for the artifact.

   Reacted by Glenn 'devalias' Grant and bcoles
4. Rhynorater commented on Aug 29, 2018

   

   Rhynorater

   on Aug 29, 2018

   Member

   More actions

   @jobertabma To swing back around on this, here is where we are at with each of your action items:

   1. Answered above
   2. Answered above
   3. The first stable version is a standard fully capable of describing a Host (as per the Host.md file)
   4. Answered above - We're getting some great community involvement!
   5. Fixed this issue - see httpUrl
   6. See above question
   7. We are implementing this per Add fields to schema: tags, notes, source #21!
   8. We're working on our use of singular and plurals. That will be resolved soon. However, we have decided to go with lowerCamelCase.

Issue actions