Description

RickGY

opened on Jul 3, 2020

Issue body actions

I've trying to use the codeql command line interface to analyze a project locally. I would like the extractor to include all the XML files, but it seems that the "codeql database create command" will only include pom.xml.

• I noticed that by including a lgtm.yml file in the repo and write the following, LGTM can include all xml files:

  extraction:
  java:
  index:
  xml_mode: all

I wonder if it's possible to do something similar while using codeql-cli?

Activity

1. RickGY

   added

   questionFurther information is requestedFurther information is requested

   on Jul 3, 2020
2. RasmusWL commented on Jul 8, 2020

   

   RasmusWL

   on Jul 8, 2020

   Member

   More actions

   Hi @RickGY, yes, instead of doing

   codeql database create --source-root <src> --language java --command <java command> <db>
   

   You can split up the database creation process to manually include the XML files:

   codeql database init --source-root=<src> --language java <db>
   codeql database trace-command --working-dir=<src> <db> <java command>
   codeql database index-files --language xml --include-extension .xml --working-dir=<src> <db>
   codeql database finalize <db>
   

   If you're just using the current directory as your source, you can skip the --working-dir <src> parts.

   Reacted by EchoLogic, Artem Smotrakov, quericy, 瓦都剋 and xxiqpl

   Reacted by EchoLogic

   Reacted by EchoLogic

   Reacted by EchoLogic and Artem Smotrakov

   Reacted by EchoLogic
3. dilanbhalla commented on Jul 8, 2020

   

   dilanbhalla

   on Jul 8, 2020

   Contributor

   More actions

   Hi @RasmusWL , I actually had a similar question. I am attempting to include xml files for a python CodeQL database I am building through the CLI, but am running into some issues. What should I include as the component in step 2 (but if I am working in python)? I am a bit unsure about what that command does, and if I run all of the commands except that step, my database includes the xml files I would like but nothing else (so none of the source code at all). Would you mind providing some guidance?
4. RasmusWL commented on Jul 9, 2020

   

   RasmusWL

   on Jul 9, 2020

   Member

   More actions

   Hi @dilanbhalla. I agree that this part is a bit tricky, and requires internal knowledge of how to do things. I was a bit confused about how to do it as well for #3911 (we're having internal talks about making it easier).

   You can use this trick to trigger the same behavior as with codeql database create 👍

   codeql database trace-command --working-dir=<src> <db>  $(codeql resolve extractor --language=python)/tools/autobuild.sh
   

   Reacted by EchoLogic

   Reacted by EchoLogic
5. RickGY

   closed this as completedon Jul 9, 2020
6. RickGY

   reopened this on Jul 9, 2020
7. RickGY commented on Jul 9, 2020

   

   RickGY

   on Jul 9, 2020

   Author

   More actions

   Thanks a lot! @RasmusWL
8. RasmusWL commented on Jul 9, 2020

   

   RasmusWL

   on Jul 9, 2020

   Member

   More actions

   No problem 😊

   Gonna close this now then
9. RasmusWL

   closed this as completedon Jul 9, 2020
10. RasmusWL commented on Jul 10, 2020

    

    RasmusWL

    on Jul 10, 2020

    Member

    More actions

    Hi Rasmus, apologies in advance for the long follow up, but I have a question unrelated to this query and am unsure how else to contact you (since our discussion thread from earlier is now closed). This may seem a little silly, but the trace-command you showed my for python won't work due to the simple error that the '$' is not recognized. My end goal is to simply use the CLI to build a python database that includes some custom xml files I wrote, so I believe your method would work for python (init, index the xml files, trace-command, finalize). Does your PR need to be merged before this trace-command will work? Or is it something simple that I may be doing wrong with regards to the expression starting with '$'? And lastly, if including the xml is not at all possible, would you happen to know any other method to include custom data (maybe through something like a csv) and reference it within a python ql file? Thank you so much!
    - #3878 (comment)

    Hi @dilanbhalla, you can comment on closed issues, no problem. I think the most appropriate place to ask questions about the CLI in the future is on https://github.com/github/codeql-cli-binaries/issues. You can always use https://github.com/github/securitylab/discussions as well for CodeQL questions 😊

    I guess the part you're having trouble with is $(codeql resolve extractor --language=python)/tools/autobuild.sh. That is just using command substitution in bash. Basically it means, run codeql resolve extractor --language=python and insert the output of that command in place of $(codeql resolve extractor --language=python). So if you're running a different shell where command substitution doesn't work, you can do the following:

    $ codeql resolve extractor --language=python
    /foo/bar/codeql/python
    $ codeql database trace-command --working-dir=<src> <db>  /foo/bar/codeql/python/tools/autobuild.sh
    

    if you're on windows, you should probably use autobuild.cmd instead of autobuild.sh 😊

    Reacted by dilanbhalla
11. dilanbhalla commented on Jul 12, 2020

    

    dilanbhalla

    on Jul 12, 2020

    Contributor

    More actions

    Hi @RasmusWL! This worked like a charm. Sorry I did not realize I can continue to comment on a closed issue and thank you so much for the advice. Everything is running perfectly for me now, with the .xml files I was attempting to include earlier. Thanks again!

    Reacted by Rasmus Wriedt Larsen

Issue actions