I am attempting to emulate the Python/C API's PyRun_InteractiveLoop() function, but from a different input system used by my employer. The Python FAQ (http://docs.python.org/3/faq/extending.html#how-do-i-tell-incomplete-input-from-invalid-input) has the following code, used to check if a given character array is a complete Python block:
#include <Python.h>
#include <node.h>
#include <errcode.h>
#include <grammar.h>
#include <parsetok.h>
#include <compile.h>
int testcomplete(char *code)
/* code should end in \n */
/* return -1 for error, 0 for incomplete, 1 for complete */
{
node *n;
perrdetail e;
n = PyParser_ParseString(code, &_PyParser_Grammar,
Py_file_input, &e);
if (n == NULL) {
if (e.error == E_EOF)
return 0;
return -1;
}
PyNode_Free(n);
return 1;
}
This gives an undeclared variable error for _PyParser_Grammar. Searching the Python source code, I did not find any headers that declared _PyParser_Grammar. Looking further, I found it referenced by a few functions, in particular meta_grammar() and Py_meta_grammar() as defined in metagrammar.c.
While meta_grammar() is defined in pgen.h, it gives an undefined symbol error despite compiling (g++ for my test code) with -lpython3.3m. A quick nm revealed that meta_grammar() is not a visible symbol in libpython3.3m.so, while Py_meta_grammar() is. However, when searching the Python source code, I have not found any header that declares Py_meta_grammar().
Am I missing something? Where are these symbols declared/defined?
After a great deal of faffing about, it seems there are no headers declaring _PyParser_Grammar. However, declaring
extern grammar _PyParser_Grammar;
in each of the C++ source files that need access works perfectly. Not particularly elegant, but at least it compiles.
Related
I use SimpleINI library on Linux. There is the following comment:
// Defines the conversion classes for different libraries. Before including
// SimpleIni.h, set the converter that you wish you use by defining one of the
// following symbols.
//
// SI_NO_CONVERSION Do not make the "W" wide character version of the
// library available. Only CSimpleIniA etc is defined.
// SI_CONVERT_GENERIC Use the Unicode reference conversion library in
// the accompanying files ConvertUTF.h/c
// SI_CONVERT_ICU Use the IBM ICU conversion library. Requires
// ICU headers on include path and icuuc.lib
// SI_CONVERT_WIN32 Use the Win32 API functions for conversion.
When I try to compile the following code:
#define SI_NO_CONVERSION
#include "SimpleIni.h"
int main()
{
CSimpleIni ini;
return 0;
}
I get the compilation error: ‘CSimpleIniA’ was not declared in this scope It looks like SI_NO_CONVERSION is not defined in SimpleIni.h. Could you explain what's going on ?
SimpleIni.h only defines SI_Case and SI_NoCase when either of SI_CONVERT_GENERIC, SI_CONVERT_ICU or SI_CONVERT_WIN32 is defined. This leaves SI_Case and SI_NoCase undefined when (only) SI_NO_CONVERSION is defined, which causes the CSimpleIniTempl template instantiation to fail, with the related compile errors. This is an oversight/bug in the library and should be reported to the author.
As a workaround, adding the missing definitions before #include "SimpleIni.h" gets the code to compile.
#define SI_NO_CONVERSION
#define SI_Case SI_GenericCase // ***
#define SI_NoCase SI_GenericNoCase // ***
#include "SimpleIni.h"
// ... etc ...
The doc you included says it all:
Only CSimpleIniA etc is defined [when SI_NO_CONVERSION is defined].
Below is an edited version of the first example in the README.md. The SI_ASSERT macro is defined in the header file but the ASSERT_EQ and ASSERT_STREQ macros, referenced in the example, are not. It's almost unforgivable that someone's examples don't compile.
As I mentioned in a comment, this is not a well-maintained project. The instructions for building and testing don't work in a very obvious way. Seems like the developer has files in his working copy that aren't in the git repository. Inexcusable.
Additionally, I get extra compilation errors when I define SI_NO_CONVERSION. Don't use this project. Use something else instead.
#include "SimpleIni.h"
int main () {
CSimpleIniA ini;
ini.SetUnicode();
SI_Error rc = ini.LoadFile("example.ini");
if (rc < 0) { /* handle error */ };
const char* pv;
pv = ini.GetValue("section", "key", "default");
ini.SetValue("section", "key", "newvalue");
pv = ini.GetValue("section", "key", "default");
}
You compile this with something like g++.
I have been recently practicing managing multiple objects and drawing them in C++ using SFML library. I wanted my textures and future resources to be more reusable so I decided to make use of Thor library which suits my needs really well.
So I've written first few lines of code based on what you can find in this tutorial and the compiler always says:
main.cpp|12|error: 'textures_holder' does not name a type
This line gives an error :
textures_holder.acquire("Dirt", thor::Resources::fromFile<sf::Texture>("Textures\\dirt_block.png"));
I'm using Code::Blocks IDE with MinGW compiler and SFML 2.5.0.
Here's my main.cpp and the header file which contains extern object :
//...
#include <Thor/Resources.hpp>
#include "Dirt_Block.h"
using namespace std;
//Adding textures to the texture library
//THIS LINE GIVES AN ERROR
textures_holder.acquire("Dirt", thor::Resources::fromFile<sf::Texture>("Textures\\dirt_block.png"));
//Rest of code...
Dirt_Block.h (only the upper part) :
#ifndef DIRT_BLOCK_H
#define DIRT_BLOCK_H
#include <SFML\Graphics.hpp>
#include <vector>
#include <Thor/Resources.hpp>
#include <Thor/Resources/SfmlLoaders.hpp>
extern sf::Vector2u screenRes;
extern thor::ResourceHolder<sf::Texture, std::string> textures_holder;
//Rest of the code
I'd like to know what is causing this error and maybe help others who may experience similiar frustrating problems. Thanks for help.
EDIT :
As suggested in the comment I've declared a few extern int variables in the Dirt_Block.h so now it looks like this :
//...
extern int test_int_up;
extern sf::Vector2u screenRes;
extern thor::ResourceHolder<sf::Texture, std::string> textures_holder;
extern int test_int_d;
//...
And then assinged to them some value in main.cpp :
//...
test_int_up = 55;
test_int_d = 55;
//Adding textures to the texture library
textures_holder.acquire("Dirt", thor::Resources::fromFile<sf::Texture>("Textures\\dirt_block.png"));
//...
But the compiler gives error :
main.cpp|9|error: 'test_int_up' does not name a type
main.cpp|10|error: 'test_int_d' does not name a type
main.cpp|12|error: 'textures_holder' does not name a type
Much less distracting to see what your problem is without all the extraneous code!
C++ programs don't start from the top of the file and run code down to the bottom. They start at the main(), and control flow proceeds from there, with one thing triggering another.
(Note: That doesn't take into account global constructor ordering, which does go in order of declaration--but you have no guarantee of the order declarations from "different files" might run in.)
Point being, you can't just make random function or method calls in the middle of a file. That's where you put declarations. You have to be inside of a function or method to make calls, e.g.
int main() {
textures_holder.acquire(
"Dirt",
thor::Resources::fromFile<sf::Texture>("Textures\\dirt_block.png")
);
...
}
I am working on a school project which requires to work with sheepdog. Sheepdog provides a c api which enables you to connect to a sheepdog server.
First i create c source file(test.c) with the following content :
#include "sheepdog/sheepdog.h"
#include <stdio.h>
int main()
{
struct sd_cluster *c = sd_connect("192.168.1.104:7000");
if (!c) {
fprintf(stderr, "failed to connect %m\n");
return -1;
}else{
fprintf(stderr, "connected successfully %m\n");
}
return 0;
}
then i compile with no error using the following command
gcc -o test test.c -lsheepdog -lpthread
But what i need is to use it with c++ project so i created a cpp file(test.cpp) with the following content :
extern "C"{
#include "sheepdog/sheepdog.h"
}
#include <stdio.h>
int main()
{
struct sd_cluster *c = sd_connect("192.168.1.104:7000");
if (!c) {
fprintf(stderr, "failed to connect %m\n");
return -1;
}else{
fprintf(stderr, "connected successfully %m\n");
}
return 0;
}
now, when i compiled using the following command :
g++ -o test test.cpp -lsheepdog -lpthread
I got this error :
You can't just wrap extern "C" around a header and expect it to compile in a C++ program. For example, the header sheepdog_proto.h uses an argument named new; that's a keyword in C++, so there's no way that will compile as C++. The library was not designed to be called from C++.
I agree with #PeteBecker. From a quick look around Google, I am not sure there is an easy solution. Sheepdog is using C features and names that don't port well to C++. You might need to hack sheepdog fairly extensively. For example:
move the inline functions out of sheepdog_proto.h into a new C file, leaving prototypes in their place. This should take care of the offsetof errors, e.g., discussed in this answer.
#define new not_a_keyword_new in sheepdog/sheepdog.h
and whatever other specific changes you have to make to get it to compile. More advice from the experts here.
As sheepdog was not designed to be useable from C++ you should build a tiny wrapper in C language to call the functions from sheepdog and only call the wrapper from your c++ code. Some hints to write such a wrapper:
void * is great to pass opaque pointers
extractors can help to access badly named members. If a struct has a member called new (of type T), you could write:
T getNew(void *otherstruct); // declaration in .h
and
T getNew(void *otherstruct) { // implementation in a c file
return ((ActualStruct *) otherstruct)->new;
}
Depending on the complexity of sheepdog (I do not know it) and the part you want to use, it may or not be an acceptable solution. But it is the way I would try facing such a problem.
Anyway, the linker allows mixing modules compiled in C and in C++, either in static linking or dynamic linking.
I am getting warnings when I am trying to include <boost/thread.hpp> in C++ Builder. For every unit I am including it, C++ Builder shows up these 2 lines:
thread_heap_alloc.hpp(59): W8128 Can't import a function being defined
thread_heap_alloc.hpp(69): W8128 Can't import a function being defined
Already tried some things, nothing worked though.
It compiles correctly, however, it's getting on my nerves. Why is this message being shown?
The lines are:
#include <boost/config/abi_prefix.hpp>
namespace boost
{
namespace detail
{
inline BOOST_THREAD_DECL void* allocate_raw_heap_memory(unsigned size)
{
void* const eap_memory=detail::win32::HeapAlloc(detail::win32::GetProcessHeap(),0,size);
if(!heap_memory)
{
throw std::bad_alloc();
}
return heap_memory;
}
inline BOOST_THREAD_DECL void free_raw_heap_memory(void* heap_memory)
{
BOOST_VERIFY(detail::win32::HeapFree(detail::win32::GetProcessHeap(),0,heap_memory)!=0);
}
where 59 is the { below the BOOST_THREAD_DECL, as is 69. Looks like BOOST_THREAD_DECL is not defined properly or mis-defined, trying to follow through the Boost code is not that easy.
This is Boost 1.39.
add #define BOOST_THREAD_USE_LIB before including the thread.hpp.
This is what I tested:
#define BOOST_THREAD_USE_LIB
extern "C"
{
namespace boost
{
void tss_cleanup_implemented( void )
{
/*
This function's sole purpose is to cause a link error in cases where
automatic tss cleanup is not implemented by Boost.Threads as a
reminder that user code is responsible for calling the necessary
functions at the appropriate times (and for implementing an a
tss_cleanup_implemented() function to eliminate the linker's
missing symbol error).
If Boost.Threads later implements automatic tss cleanup in cases
where it currently doesn't (which is the plan), the duplicate
symbol error will warn the user that their custom solution is no
longer needed and can be removed.*/
}
}
}
#include <boost/thread.hpp>
Then set 'Link with Dynamic RTL' and 'Link with Runtime Packages'.
This does a clean build and starts a thread properly.
Is it at all possible to parse C++ with incomplete declarations with clang with its existing libclang API ? I.e. parse .cpp file without including all the headers, deducing declarations on the fly. so, e.g. The following text:
A B::Foo(){return stuff();}
Will detect unknown symbol A, call my callback that deducts A is a class using my magic heuristic, then call this callback the same way with B and Foo and stuff. In the end I want to be able to infer that I saw a member Foo of class B returning A, and stuff is a function.. Or something to that effect.
context: I wanna see if I can do sensible syntax highlighting and on the fly code analysis without parsing all the headers very quickly.
[EDIT] To clarify, I'm looking for very heavily restricted C++ parsing, possibly with some heuristic to lift some of the restrictions.
C++ grammar is full of context dependencies. Is Foo() a function call or a construction of a temporary of class Foo? Is Foo<Bar> stuff; a template Foo<Bar> instantiation and declaration of variable stuff, or is it weird-looking 2 calls to overloaded operator < and operator > ? It's only possible to tell in context, and context often comes from parsing the headers.
What I'm looking for is a way to plug my custom convention rules. E.g. I know that I don't overload Win32 symbols, so I can safely assume that CreateFile is always a function, and I even know its signature. I also know that all my classes start with a capital letter and are nouns, and functions are usually verbs, so I can reasonably guess that Foo and Bar are class names. In a more complex scenario, I know I don't write side-effect-free expressions like a < b > c; so I can assume that a is always a template instantiation. And so on.
So, the question is whether it's possible to use Clang API to call back every time it encounters an unknown symbol, and give it an answer using my own non-C++ heuristic. If my heuristic fails, then the parse fails, obviously. And I'm not talking about parsing Boost library :) I'm talking about very simple C++, probably without templates, restricted to some minimum that clang can handle in this case.
I know the question is fairly old, but have a look here :
LibFuzzy is a library for heuristically parsing C++ based on Clang's
Lexer. The fuzzy parser is fault-tolerant, works without knowledge of
the build system and on incomplete source files. As the parser
necessarily makes guesses, the resulting syntax tree may be partially
wrong.
It is a sub-project from clang-highlight, an (experimental?) tool which seems to be no longer developed.
I'm only interested in the fuzzy parsing part and forked it on my github page where I fixed several minor issues and made the tool autonomous (it can be compiled outside clang's source tree). Don't try to compile it with C++14 (which G++ 6's default mode), because there will be conflicts with make_unique.
According to this page, clang-format has its own fuzzy parser (and is actively developed), but the parser was (is ?) more tighly coupled to the tool.
Unless you heavily restrict the code that people are allowed to write, it is basically impossible to do a good job of parsing C++ (and hence syntax highlighting beyond keywords/regular expressions) without parsing all the headers. The pre-processor is particularly good at screwing things up for you.
There are some thoughts on the difficulties of fuzzy parsing (in the context of visual studio) here which might be of interest: http://blogs.msdn.com/b/vcblog/archive/2011/03/03/10136696.aspx
Another solution which I think will suit more the OP than fuzzy parsing.
When parsing, clang maintains Semantic information through the Sema part of the analyzer. When encountering an unknown symbol, Sema will fallback to ExternalSemaSource to get some information about this symbol. Through this, you could implement what you want.
Here is a quick example how to set up it. It is not entirely functional (I'm not doing anything in the LookupUnqualified method), you might need to do further investigations and I think it is a good start.
// Declares clang::SyntaxOnlyAction.
#include <clang/Frontend/FrontendActions.h>
#include <clang/Tooling/CommonOptionsParser.h>
#include <clang/Tooling/Tooling.h>
#include <llvm/Support/CommandLine.h>
#include <clang/AST/AST.h>
#include <clang/AST/ASTConsumer.h>
#include <clang/AST/RecursiveASTVisitor.h>
#include <clang/Frontend/ASTConsumers.h>
#include <clang/Frontend/FrontendActions.h>
#include <clang/Frontend/CompilerInstance.h>
#include <clang/Tooling/CommonOptionsParser.h>
#include <clang/Tooling/Tooling.h>
#include <clang/Rewrite/Core/Rewriter.h>
#include <llvm/Support/raw_ostream.h>
#include <clang/Sema/ExternalSemaSource.h>
#include <clang/Sema/Sema.h>
#include "clang/Basic/DiagnosticOptions.h"
#include "clang/Frontend/TextDiagnosticPrinter.h"
#include "clang/Frontend/CompilerInstance.h"
#include "clang/Basic/TargetOptions.h"
#include "clang/Basic/TargetInfo.h"
#include "clang/Basic/FileManager.h"
#include "clang/Basic/SourceManager.h"
#include "clang/Lex/Preprocessor.h"
#include "clang/Basic/Diagnostic.h"
#include "clang/AST/ASTContext.h"
#include "clang/AST/ASTConsumer.h"
#include "clang/Parse/Parser.h"
#include "clang/Parse/ParseAST.h"
#include <clang/Sema/Lookup.h>
#include <iostream>
using namespace clang;
using namespace clang::tooling;
using namespace llvm;
class ExampleVisitor : public RecursiveASTVisitor<ExampleVisitor> {
private:
ASTContext *astContext;
public:
explicit ExampleVisitor(CompilerInstance *CI, StringRef file)
: astContext(&(CI->getASTContext())) {}
virtual bool VisitVarDecl(VarDecl *d) {
std::cout << d->getNameAsString() << "#\n";
return true;
}
};
class ExampleASTConsumer : public ASTConsumer {
private:
ExampleVisitor visitor;
public:
explicit ExampleASTConsumer(CompilerInstance *CI, StringRef file)
: visitor(CI, file) {}
virtual void HandleTranslationUnit(ASTContext &Context) {
// de cette façon, on applique le visiteur sur l'ensemble de la translation
// unit
visitor.TraverseDecl(Context.getTranslationUnitDecl());
}
};
class DynamicIDHandler : public clang::ExternalSemaSource {
public:
DynamicIDHandler(clang::Sema *Sema)
: m_Sema(Sema), m_Context(Sema->getASTContext()) {}
~DynamicIDHandler() = default;
/// \brief Provides last resort lookup for failed unqualified lookups
///
/// If there is failed lookup, tell sema to create an artificial declaration
/// which is of dependent type. So the lookup result is marked as dependent
/// and the diagnostics are suppressed. After that is's an interpreter's
/// responsibility to fix all these fake declarations and lookups.
/// It is done by the DynamicExprTransformer.
///
/// #param[out] R The recovered symbol.
/// #param[in] S The scope in which the lookup failed.
virtual bool LookupUnqualified(clang::LookupResult &R, clang::Scope *S) {
DeclarationName Name = R.getLookupName();
std::cout << Name.getAsString() << "\n";
// IdentifierInfo *II = Name.getAsIdentifierInfo();
// SourceLocation Loc = R.getNameLoc();
// VarDecl *Result =
// // VarDecl::Create(m_Context, R.getSema().getFunctionLevelDeclContext(),
// // Loc, Loc, II, m_Context.DependentTy,
// // /*TypeSourceInfo*/ 0, SC_None, SC_None);
// if (Result) {
// R.addDecl(Result);
// // Say that we can handle the situation. Clang should try to recover
// return true;
// } else{
// return false;
// }
return false;
}
private:
clang::Sema *m_Sema;
clang::ASTContext &m_Context;
};
// *****************************************************************************/
LangOptions getFormattingLangOpts(bool Cpp03 = false) {
LangOptions LangOpts;
LangOpts.CPlusPlus = 1;
LangOpts.CPlusPlus11 = Cpp03 ? 0 : 1;
LangOpts.CPlusPlus14 = Cpp03 ? 0 : 1;
LangOpts.LineComment = 1;
LangOpts.Bool = 1;
LangOpts.ObjC1 = 1;
LangOpts.ObjC2 = 1;
return LangOpts;
}
int main() {
using clang::CompilerInstance;
using clang::TargetOptions;
using clang::TargetInfo;
using clang::FileEntry;
using clang::Token;
using clang::ASTContext;
using clang::ASTConsumer;
using clang::Parser;
using clang::DiagnosticOptions;
using clang::TextDiagnosticPrinter;
CompilerInstance ci;
ci.getLangOpts() = getFormattingLangOpts(false);
DiagnosticOptions diagnosticOptions;
ci.createDiagnostics();
std::shared_ptr<clang::TargetOptions> pto = std::make_shared<clang::TargetOptions>();
pto->Triple = llvm::sys::getDefaultTargetTriple();
TargetInfo *pti = TargetInfo::CreateTargetInfo(ci.getDiagnostics(), pto);
ci.setTarget(pti);
ci.createFileManager();
ci.createSourceManager(ci.getFileManager());
ci.createPreprocessor(clang::TU_Complete);
ci.getPreprocessorOpts().UsePredefines = false;
ci.createASTContext();
ci.setASTConsumer(
llvm::make_unique<ExampleASTConsumer>(&ci, "../src/test.cpp"));
ci.createSema(TU_Complete, nullptr);
auto &sema = ci.getSema();
sema.Initialize();
DynamicIDHandler handler(&sema);
sema.addExternalSource(&handler);
const FileEntry *pFile = ci.getFileManager().getFile("../src/test.cpp");
ci.getSourceManager().setMainFileID(ci.getSourceManager().createFileID(
pFile, clang::SourceLocation(), clang::SrcMgr::C_User));
ci.getDiagnosticClient().BeginSourceFile(ci.getLangOpts(),
&ci.getPreprocessor());
clang::ParseAST(sema,true,false);
ci.getDiagnosticClient().EndSourceFile();
return 0;
}
The idea and the DynamicIDHandler class are from cling project where unknown symbols are variable (hence the comments and the code).
OP doesn't want "fuzzy parsing". What he wants is full context-free parsing of the C++ source code, without any requirement for name and type resolution.
He plans to make educated guesses about the types based on the result of the parse.
Clang proper tangles parsing and name/type resolution, which means it must have all that background type information available when it parses. Other answers suggest a LibFuzzy that produces incorrect parse trees, and some fuzzy parser for clang-format about which I know nothing. If one insists on producing a classic AST, none of these solutions will produce the "right" tree in the face of ambiguous parses.
Our DMS Software Reengineering Toolkit with its C++ front end can parse C++ source without the type information, and produces accurate "ASTs"; these are actually abstract syntax dags where forks in trees represent different possible interpretations of the source code according to a language-precise grammar (ambiguous (sub)parses).
What Clang tries to do is avoid producing these multiple sub-parses by using type information as it parses. What DMS does is produce the ambiguous parses, and in (optional) post-parsing (attribute-grammar-evaluation) pass, collect symbol table information and eliminate the sub-parses which are inconsistent with the types; for well-formed programs, this produces a plain AST with no ambiguities left.
If OP wants to make heuristic guesses about the type information, he will need to know these possible interpretations. If they are eliminated in advance, he cannot straightforwardly guess what types might be needed. An interesting possibility is the idea of modifying the attribute grammar (provided in source form as part of the DMS's C++ front end), which already knows all of C++ type rules, to do this with partial information. That would be an enormous head start over building the heuristic analyzer from scratch, given that it has to know about 600 pages of arcane name and type resolution rules from the standard.
You can see examples of the (dag) produced by DMS's parser.