I am trying to use the readText function:
import std.stdio;
import std.file;
string xmlName = r"D:\files\123.xml";
File file;
void main()
{
writeln("Edit source/app.d to start your project.");
file = File(xmlName, "r");
string file_text = file.readText;
}
I am getting an error:
Error: template std.file.readText cannot deduce function from argument types !()(File), candidates are:
C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d(499,3): std.file.readText(S = string, R)(auto ref R name) if (isSomeString!S && (isInputRange!R && !isInfinite!R && isSomeChar!(ElementType!R) || is(StringTypeOf!R)))
What am I doing wrong?
readText takes a string argument, being the file name to read from. Since you've opened the file with File(xmlName, "r"), you should be using the methods defined in std.stdio.File.
It seems what you want is to read the entirety of the file contents into a string. In this case, I suggest replacing the two last lines in your main function with string file_text = readText(xmlName);
Since you want to use the std.file's readText(), you need to slighly modify your code:
import std.stdio;
import std.file;
string xmlName = "D:/files/123.xml";
void main() {
writeln("Edit source/app.d to start your project.");
string file_text = file.readText(xmlName);
}
Notice that we no longer create an instance of a File as it is not needed here...
Related
Currently I have the following two proto definitions, both .proto files are in same folder:
topmessage.proto:
syntax = "proto3";
message TopMessage {
// some fields
}
crestmessage.proto:
syntax = "proto3";
import "topmessage.proto";
message CrestMessage {
// some fields
TopMessage tm = 4;
}
Then, as part of my reader.cc, I am trying to build a file descriptor for the proto definition that user passes in. More specifically, the user will pass the path of the .proto file in as an argument, and the program will then read the file and build a file descriptor from it. Here is how this function is implemented, it mainly follows the blog by Floris https://vdna.be/site/index.php/2016/05/google-protobuf-at-run-time-deserialization-example-in-c/:
DescriptorPool pool;
const FileDescriptor* buildFileDescriptor(string file_path, string file_name) {
int def_messageFile = open(file_path.c_str(), O_RDONLY);
FileInputStream file_input(def_messageFile);
Tokenizer input(&file_input, NULL);
FileDescriptorProto file_desc_proto;
Parser parser;
if (!parser.Parse(&input, &file_desc_proto)) {
cerr << "Failed to parse .proto definition:" << endl;
return NULL;
}
file_desc_proto.set_name(file_name);
const FileDescriptor* file_desc = pool.BuildFile(file_desc_proto);
return file_desc;
}
The question now is when I am trying to build CrestMessage, and its proto definition file is also the one passed in as argument. For simplicity, I first build TopMessage by manually providing its file address since it is the dependency, and it works fine, and I can also find TopMessage in the pool, which is global. However, when I try to build CrestMessage, it gives the error.
const FileDescriptor* file_desc2 = buildFileDescriptor("topmessage.proto", "TopMessage");
cout << pool.FindFileByName("TopMessage") << endl;
const FileDescriptor* file_desc = buildFileDescriptor(definition_path, "CrestMessage");
cout << file_desc->name() << endl;
enter image description here
I have not find anything in the api description by Google that says how to import, does anyone have ideas on what should be used?
it's a name problem, instead of using message name when building FileDescriptor, use the name of the .proto file ("topmessage.proto" for example)
I'm having trouble reading other Stack Overflow posts so after a few hours I'm looking for help.
I have two methods that I want to test. And I'd like to test the second one using Mock, but having trouble figuring out what to do.
Here's the first method:
String readFileContents(Path filePath) {
StringBuilder fileContents = new StringBuilder()
BufferedReader br = Files.newBufferedReader(filePath, StandardCharsets.UTF_8)
String line
while ((line = br.readLine()) != null) {
fileContents.append(line).append('\n')
}
fileContents
}
And I test it with
class CdmFileSpec extends Specification {
private CdmFile cdmFile
private static final String filePath = 'src/test/resources/cdm/test/cdmFileTestFile.txt'
void setup() {
cdmFile = new CdmFile()
}
void 'test noFileExists'() {
given:
Path notRealPath = Paths.get('src/test/resources/cdm//test/notreal.txt')
when:
String fileContents = cdmFile.readFileContents(notRealPath)
then:
thrown NoSuchFileException
}
void 'test readFileContents() reads file contents'() {
given:
Path testFilePath = Paths.get(filePath)
when:
String fileContents = cdmFile.readFileContents(testFilePath)
then:
fileContents.contains('hip hop horrayy\n\nhoooo\n\nheyyy\n\nhoooo')
}
}
This works as I've placed a real file in the filePath.
I'm wondering... how can I test the next method using Mock?
void eachLineInFileAsString(Path filePath,
#ClosureParams(value = SimpleType, options = ['java.lang.String'] )Closure applyLine) {
BufferedReader br = Files.newBufferedReader(filePath)
String line
while ((line = br.readLine()) != null) {
applyLine.call(line)
}
}
The problem with mocking in so many cases is that methods create their own dependencies instead of having them injected or calling a mockable service method creating them. I suggest you refactor your code just a little bit, extracting BufferedReader creation into a service method:
package de.scrum_master.stackoverflow.q56772468
import groovy.transform.stc.ClosureParams
import groovy.transform.stc.SimpleType
import java.nio.charset.StandardCharsets
import java.nio.file.Files
import java.nio.file.Path
class CdmFile {
String readFileContents(Path filePath) {
StringBuilder fileContents = new StringBuilder()
BufferedReader br = createBufferedReader(filePath)
String line
while ((line = br.readLine()) != null) {
fileContents.append(line).append('\n')
}
fileContents
}
void eachLineInFileAsString(
Path filePath,
#ClosureParams(value = SimpleType, options = ['java.lang.String']) Closure applyLine
) {
BufferedReader br = createBufferedReader(filePath)
String line
while ((line = br.readLine()) != null) {
applyLine.call(line)
}
}
protected BufferedReader createBufferedReader(Path filePath) {
Files.newBufferedReader(filePath, StandardCharsets.UTF_8)
}
}
Now mocking is quite simple and you don't even need your test resource file anymore (only if you want to do an integration test without mocks):
package de.scrum_master.stackoverflow.q56772468
import spock.lang.Specification
import java.nio.charset.StandardCharsets
import java.nio.file.NoSuchFileException
import java.nio.file.Path
import java.nio.file.Paths
class CmdFileTest extends Specification {
private static final String filePath = 'mock/cdmTestFile.txt'
private static final String fileContent = """
I heard, that you're settled down
That you found a girl and you're, married now
I heard, that your dreams came true
I guess she gave you things
I didn't give to you
""".stripIndent()
private CdmFile cdmFile
void setup() {
cdmFile = Spy() {
createBufferedReader(Paths.get(filePath)) >> {
new BufferedReader(
new InputStreamReader(
new ByteArrayInputStream(
fileContent.getBytes(StandardCharsets.UTF_8)
)
)
)
}
}
}
def "non-existent file leads to exception"() {
given:
Path notRealPath = Paths.get('notreal.txt')
when:
cdmFile.readFileContents(notRealPath)
then:
thrown NoSuchFileException
}
def "read file contents into a string"() {
given:
Path testFilePath = Paths.get(filePath)
when:
String fileContents = cdmFile.readFileContents(testFilePath)
then:
fileContents.contains("your dreams came true\nI guess")
}
def "handle file content line by line"() {
given:
def result = []
def closure = { line -> result << line }
Path testFilePath = Paths.get(filePath)
when:
cdmFile.eachLineInFileAsString(testFilePath, closure)
then:
result == fileContent.split("\n")
}
}
Please note that I am using a Spy() here, i.e. leaving the original CdmFile object intact and just stubbing the service method createBufferedReader(..) when called with exactly parameter Paths.get(filePath). For other paths the original method is called, which is important for the non-existent file test or if you want to add tests involving real resource file loading like in your own example.
Whenever it is difficult to test a class or component, difficult to inject mocks or otherwise isolate the subject under test, that is a reason to refactor your application code for better testability. When done right also it should also result in better separation of concerns and better componentisation. If your tests become very sophisticated, contrived, brittle and hard to understand and maintain, that is usually a smell and you ought to refactor the application code instead.
There's no need for a Mock, as you can just use a locally defined Closure:
def "test the method"() {
given:
def result = []
def closure = { line -> result << line }
Path testFilePath = Paths.get(filePath)
when:
eachLineInFileAsString(testFilePath, closure)
then: // I'm guessing here
result == [
'line 1',
'line 2',
'line 3',
'line 4'
]
}
I need to substitute characters of a tuple using Pig UDF. For eg, if i have a line in the file as "hello world, Hello WORLD, hello\WORLD" required to be transformed as "hello_world,hello_world,hello_world". To accomplish this, i tried below UDF:
package myUDF;
import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.TupleFactory;
public class ReplaceValues extends EvalFunc<Tuple>
{
public Tuple exec(Tuple input) throws IOException {
if (input == null || input.size() == 0)
return null;
try{
String str = (String)input.get(0);
str=str.replace(" ", "_");
str=str.replace("/","");
str=str.replace("\\","");
TupleFactory tf = TupleFactory.getInstance();
Tuple t = tf.newTuple();
t.append(str);
return t;
}catch(Exception e){
throw new IOException("Caught exception processing input row ", e);
}
}
}
but when calling this UDF via pig script i am facing issues, please help me in resolving this:
A = load '/user/cloudera/Stage/ActualDataSet.csv' using PigStorage(',') AS (Rank:chararray,NCTNumber:chararray,Title:chararray,Recruitment:chararray);
B = FILTER A by Rank == 'Rank';
C = FOREACH B GENERATE PigUDF.ReplaceValues(B);
Error: Pig script failed to parse:
Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
You have to pass the field that you are trying to modify and not the relation B.Assuming the field that you are trying to match is Title, then you would call the UDF like below
C = FOREACH B GENERATE B.Rank,B.NCTNumber,PigUDF.ReplaceValues(B.Title),B.Recruitment;
Note that if you are trying to replace it in the entire record then your load statement is incorrect.You will have to load the entire record as one line:chararray and then pass the line to your UDF.
Also, instead of an UDF you can use REGEX to match and replace the string of your choice.
In your Pig script, you are passing entire Bag "B" in the UDF, while it accepts tuple as an argument.
Instead pass the field like B.Title as given below.
C = FOREACH B GENERATE PigUDF.ReplaceValues(B.Title);
you can call this UDF on other fields also in the same line as:
C = FOREACH B GENERATE PigUDF.ReplaceValues(B.Title), PigUDF.ReplaceValues(B.Rank);
I've abstracted a very simple situation here, in which I want to pass a list of strings into my cleanLines function and get a list of strings back out. Unfortunately, I'm new to Groovy and I've spent about a day trying to get this to work with no avail. Here's a stand-alone test that exhibits the problem I'm having:
import static org.junit.Assert.*;
import java.util.List;
import org.junit.Test;
class ConfigFileTest {
private def tab = '\t'
private def returnCarriage = '\r'
private def equals = '='
List <String> cleanLines(List <String> lines) {
lines = lines.collect(){it.findAll(){c -> c != tab && c != returnCarriage}}
lines = lines.findAll(){it.contains(equals)}
lines = lines.collect{it.trim()}
}
#Test
public void test() {
List <String> dirtyLines = [" Colour=Red",
"Shape=Square "]
List <String> cleanedLines = ["Colour=Red",
"Shape=Square"]
assert cleanLines(dirtyLines) == cleanedLine
}
}
I believe that I've followed the correct usage for collect(), findAll() and trim(). But when I run the test, it crashes on the trim() line stating
groovy.lang.MissingMethodException: No signature of method:
java.util.ArrayList.trim() is applicable for argument types: ()
values: []
. Something's suspicious.
I've been staring at this for too long and noticed that my IDE thinks the type of my first lines within the cleanLines function is List<String>, but that by the second line it has type Collection and by the third it expects type List<Object<E>>. I think that String is an Object and so this might be okay, but it certainly hints at a misunderstanding on my part. What am I doing wrong? How can I get my test to pass here?
Here's a corrected script:
import groovy.transform.Field
#Field
def tab = '\t'
#Field
def returnCarriage = '\r'
#Field
def equals = '='
List <String> cleanLines(List <String> lines) {
lines = lines.findAll { it.contains(equals) }
lines = lines.collect { it.replaceAll('\\s+', '') }
lines = lines.collect { it.trim() }
}
def dirtyLines = [" Colour=Red",
"Shape=Square "]
def cleanedLines = ["Colour=Red", "Shape=Square"]
assert cleanLines(dirtyLines) == cleanedLines
In general findAll and collect are maybe not mutually exclusive but have different purposes. Use findAll to find elements that matches certain criteria, whereas collect when you need to process/transform the whole list.
this line
lines = lines.collect(){it.findAll(){c -> c != tab && c != returnCarriage}}`
replaces the original list of Strings with the list of lists. Hence NSME for ArrayList.trim(). You might want to replace findAll{} with find{}
You can clean the lines like this:
def dirtyLines = [" Colour=Red", "Shape=Square "]
def cleanedLines = ["Colour=Red", "Shape=Square"]
assert dirtyLines.collect { it.trim() } == cleanedLines
If you separate the first line of cleanLines() into two separate lines and print lines after each, you will see the problem.
it.findAll { c -> c != tab && c != returnCarriage }
will return a list of strings that match the criteria. The collect method is called on every string in the list of lines. So you end up with a list of lists of strings. I think what you are looking for is something like this:
def cleanLines(lines) {
return lines.findAll { it.contains(equals) }
.collect { it.replaceAll(/\s+/, '') }
}
I am running following command
TestData = FOREACH records Generate From as from, MsgId as Msg, REPLACE(toAddress,';' , ',');
I am getting following error
mismatched character
'' expecting ''' 2014-04-14 12:27:56,863 [main] ERROR
org.apache.pig.tools.grunt.Grunt - ERROR 1200: mismatched character ''
expecting '''
May be it is because of ; character ? if it so then how to apply the patch for it ..
It seems to me that Pig 0.11.0-cdh4.3.0 doesn't include PIG-2507.
You either need to patch and rebuild Pig to make it work (Download the patch from here: https://issues.apache.org/jira/secure/attachment/12571848/PIG_2507.patch) or as a workaround you can create a custom UDF based on org.apache.pig.builtin.REPLACE :
E.g:
package com.example;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.pig.EvalFunc;
import org.apache.pig.FuncSpec;
import org.apache.pig.PigWarning;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.DataType;
import org.apache.pig.impl.logicalLayer.schema.Schema;
import org.apache.pig.impl.logicalLayer.FrontendException;
public class MyReplace extends EvalFunc<String> {
private String searchString;
public MyReplace(String searchString) {
this.searchString = searchString;
}
#Override
public String exec(Tuple input) throws IOException {
if (input == null || input.size() < 2)
return null;
try {
String source = (String) input.get(0);
String replacewith = (String) input.get(1);
return source.replaceAll(searchString, replacewith);
}
catch (Exception e) {
warn("Failed to process input; error - " + e.getMessage(), PigWarning.UDF_WARNING_1);
return null;
}
}
#Override
public Schema outputSchema(Schema input) {
return new Schema(new Schema.FieldSchema(null, DataType.CHARARRAY));
}
/* (non-Javadoc)
* #see org.apache.pig.EvalFunc#getArgToFuncMapping()
*/
#Override
public List<FuncSpec> getArgToFuncMapping() throws FrontendException {
List<FuncSpec> funcList = new ArrayList<FuncSpec>();
Schema s = new Schema();
s.add(new Schema.FieldSchema(null, DataType.CHARARRAY));
s.add(new Schema.FieldSchema(null, DataType.CHARARRAY));
funcList.add(new FuncSpec(this.getClass().getName(), s));
return funcList;
}
}
Pack it in a jar then you can use it:
register '/path/to/my.jar';
DEFINE myReplace com.example.MyReplace(';');
A = load 'data' as (a:chararray);
B = FOREACH A generate myReplace(a,',');
...
It can be achieved by using Pig built in function as well.
Instead of special chars in replace function call, use Unicode escape sequences.
For problem mentioned above use:
REPLACE (toAddress,'\\\\u003b','\u002c')
in place of:
REPLACE(toAddress,';' , ',')