I am trying to move some old code that used the older google-api-client gem to the Idiomatic Ruby client google-cloud-ruby.
The process is a simple query job that saves it's results to another table. In the older gem, I used a config like this:
config= {
"jobReference": {
"projectId": GOOGLE_PROJECT,
'location'=> 'europe-west2'
},
'configuration'=> {
'query'=> {
'allowLargeResults' => true,
'createDisposition' => 'CREATE_IF_NEEDED',
'writeDisposition' => 'WRITE_TRUNCATE',
'query' => sql,
'destinationTable'=> {
'projectId'=> GOOGLE_PROJECT,
'datasetId'=> 'my_dataset',
'tableId'=> table,
'location'=> 'europe-west2'
}
}
},
}
Following the docs for the newer library, I am running this as a basic test (the sql is defined elsewhere)
bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset('my_dataset')
puts(dataset.location)
puts("1")
job = bigquery.query_job(sql, table: dataset.table(table), write: 'truncate', create: 'needed')
puts("2")
job.wait_until_done!
puts("3")
job.done?
This gets as far as the puts 2, failing on job.wait_until_done! with the error Google::Cloud::NotFoundError: notFound: Not found: Job my_project:job_hApg5h0NQQb4Xbv7Sr3zzIXm5RWF
If I 'puts' the job.job_id I see the same ID as it's saying it can't find. I've tried running this in datasets in multi-region and single location and still the same error. Ultimately, I need this to run on the 'europe-west2' region only.
Can anyone help and/or point me to a working example? Thanks in advance!
As suggested by #Tlaquetzal, you can replace your SQL query to a simple SELECT 1 as below sample query and see the results.
sql = "SELECT 1 FROM `project.dataset.table`"
I'm trying to get a subset of adsets with the corresponding insights for a specific daterange (2015-11-01 to 2015-11-30)
my_campaign_id/?fields=campaign,insights.fields(impressions)
.since(1463752380).until(1463752380)
but I get back
"insights": {
"data": [
{
"impressions": "1470",
"date_start": "2015-11-30",
"date_stop": "2015-12-01"
}
],
which is wrong because it's counting impressions for Dec 1st too, in fact date_stop is "2015-12-01" and not "2015-11-30".
What't the correct way of filtering expanded fields? I couldn't find any other way in the documentation/forums.
The documentation under time_range says that unix timestamp is not supported. You have to use isoformat instead, referring to the local time of you timezone. But I have to admit I have seen an error message that stated that the time had to be defined in isoformat OR unix time.
https://developers.facebook.com/docs/marketing-api/insights/parameters/v2.6
Anyway, try this:
<campaign_id>/insights?fields=impressions&time_range[since]=2015-11-01&time_range[until]=2015-11-30
I started my MongoDB journey with the follow questions:
How to list all databases in MongoDB?
How to list all collections in a specific database?
How to list the number of documents in a specific collection?
The answers I found are as below:
show dbs
use <db name>
show collections
use <db name>
db.<collection name>.count()
All commands above were run in MongoDB shell manually. I would like to find a way to run these commands in a batch mode and get the results formatted like:
Would anyone like to share any ideas implementing it?
Any specific snippet would be greatly appreciated.
Well you can basically just create a JavaScript file and run it through the mongo shell if you want:
var db = db.getSiblingDB("admin");
var output = [];
db.runCommand({ listDatabases: 1 }).databases.map(
function(x){ return x.name }
).forEach(function(dbname) {
var cdb = db.getSiblingDB(dbname);
cdb.getCollectionNames().forEach(function(coll) {
var count = cdb.getCollection(coll).count();
output.push({
db: cdb,
collection: coll,
count: count
});
})
});
printjson( output );
And then just invoke it. I called the file "lister.js":
mongo lister.js
So you can use the print() or printjson() methods that are available to the shell to produce output how you want.
Otherwise consult the MongoDB driver for your favorite scripting language and implement something along the lines of what is shown above. All the basic methods and principles will be available.
In the book The Art of Unit Testing it talks about wanting to create maintainable and readable unit tests. Around page 204 it mentions that one should try to avoid multiple asserts in one test and, perhaps compare objects with an overridden Equals method. This works great when we have only one object to compare the expected vs. actual results. However what if we have a list (or collection) of said objects.
Consider the test below. I have more than one assert. In fact, there are two separate loops calling asserts. In this case I will end up with 5 assertions. 2 to check the contents of one list exist in another, and 2 vice versa. The 5th comparing the number of elements in the lists.
If anyone has suggestions to improve this test, I'm all ears. I am using MSTest at the moment, though I replaced MSTest's Assert with NUnits for the fluent API (Assert.That).
Current Refactored Code:
[TestMethod]
#if !NUNIT
[HostType("Moles")]
#else
[Moled]
#endif
public void LoadCSVBillOfMaterials_WithCorrectCSVFile_ReturnsListOfCSVBillOfMaterialsThatMatchesInput()
{
//arrange object(s)
var filePath = "Path Does Not Matter Because of Mole in File object";
string[] csvDataCorrectlyFormatted = { "1000, 1, Alt 1, , TBD, 1, 10.0, Notes, , Description, 2.50, ,A",
"1001, 1, Alt 2, , TBD, 1, 10.0, Notes, , Description, 2.50, ,A" };
var materialsExpected = new List<CSVMaterial>();
materialsExpected.Add(new CSVMaterial("1000", 1, "Alt 1", "TBD", 1m, 10.0m, "Notes", "Description", 2.50m,"A"));
materialsExpected.Add(new CSVMaterial("1001", 1, "Alt 2", "TBD", 1m, 10.0m, "Notes", "Description", 2.50m,"A"));
//by-pass actually hitting the file system and use in-memory representation of CSV file
MFile.ReadAllLinesString = s => csvDataCorrectlyFormatted;
//act on object(s)
var materialsActual = modCSVImport.LoadCSVBillOfMaterials(filePath);
//assert something happended
Assert.That(materialsActual.Count,Is.EqualTo(materialsExpected.Count));
materialsExpected.ForEach((anExpectedMaterial) => Assert.That(materialsActual.Contains(anExpectedMaterial)));
materialsActual.ForEach((anActualMaterial) => Assert.That(materialsExpected.Contains(anActualMaterial)));
}
Original Multi-Assert Unit-Test:
...
//1st element
Assert.AreEqual("1000", materials[0].PartNumber);
Assert.AreEqual(1, materials[0].SequentialItemNumber);
Assert.AreEqual("Alt 1", materials[0].AltPartNumber);
Assert.AreEqual("TBD", materials[0].VendorCode);
Assert.AreEqual(1m, materials[0].Quantity);
Assert.AreEqual(10.0m, materials[0].PartWeight);
Assert.AreEqual("Notes", materials[0].PartNotes);
Assert.AreEqual("Description", materials[0].PartDescription);
Assert.AreEqual(2.50m, materials[0].UnitCost);
Assert.AreEqual("A", materials[1].Revision);
//2nd element
Assert.AreEqual("1001", materials[1].PartNumber);
Assert.AreEqual(1, materials[1].SequentialItemNumber);
Assert.AreEqual("Alt 2", materials[1].AltPartNumber);
Assert.AreEqual("TBD", materials[1].VendorCode);
Assert.AreEqual(1m, materials[1].Quantity);
Assert.AreEqual(10.0m, materials[1].PartWeight);
Assert.AreEqual("Notes", materials[1].PartNotes);
Assert.AreEqual("Description", materials[1].PartDescription);
Assert.AreEqual(2.50m, materials[1].UnitCost);
Assert.AreEqual("A", materials[1].Revision);
}
I frequently have more than one assertion. If it's all part of testing one logical unit of work, I don't see any problem with that.
Now, I do agree that if you've got a type which overrides Equals, that makes tests much simpler than your second form. But in your first test, it looks you really just want to assert that the resulting collection equals an expected one. I think that's logically one assertion - it's just that currently you're performing multiple mini-assertions to test it.
Some unit test frameworks have methods to test whether two collections are equal - and if the one you're using doesn't, you can easily write one. I recently did exactly this in my "reimplementing LINQ to Objects" blog series, because although NUnit provides a helper method, its diagnostics aren't terribly helpful. I refactored the code from MoreLINQ very slightly, basically.
This is the refactoring I'm currently using. I overrode the ToString() method of CSVMaterial and added a more useful assert message. So I think this helps with code readability and maintainability. It also makes the unit test trustworthy (due to the helpful diagnostic message).
And Jon, thanks for the thought about a logical unit of work. My refactored code does about the same thing as the previous iteration. Both still test one logical thing. Also, I'll have to look into the MoreLINQ stuff. If it's in your C# InDepth 2nd edition book, I'll come across it as I bought the MEAP version from Manning. Thanks for your help.
public void LoadCSVBillOfMaterials_WithCorrectCSVFile_ReturnsListOfCSVBillOfMaterialsThatMatchesInput()
{
//arrange object(s)
var filePath = "Path Does Not Matter Because of Mole in File object";
string[] csvDataCorrectlyFormatted = { "1000, 1, Alt 1, , TBD, 1, 10.0, Notes, , Description, 2.50, ,A",
"1001, 1, Alt 2, , TBD, 1, 10.0, Notes, , Description, 2.50, ,A" };
var materialsExpected = new List<CSVMaterial>();
materialsExpected.Add(new CSVMaterial("1001", 1, "Alt 1", "TBD", 1m, 10.0m, "Notes", "Description", 2.50m,"A"));
materialsExpected.Add(new CSVMaterial("1001", 1, "Alt 2", "TBD", 1m, 10.0m, "Notes", "Description", 2.50m,"A"));
//by-pass actually hitting the file system and use in-memory representation of CSV file
MFile.ReadAllLinesString = s => csvDataCorrectlyFormatted;
//act on object(s)
var materialsActual = modCSVImport.LoadCSVBillOfMaterials(filePath);
//assert something happended
//Setup message for failed asserts
var assertMessage = new StringBuilder();
assertMessage.AppendLine("Actual Materials:");
materialsActual.ForEach((m) => assertMessage.AppendLine(m.ToString()));
assertMessage.AppendLine("Expected Materials:");
materialsExpected.ForEach((m) => assertMessage.AppendLine(m.ToString()));
Assert.That(materialsActual, Is.EquivalentTo(materialsExpected),assertMessage.ToString());
}
I'm writing a Multiplayer C++ based game.
I need a flexible file format to store information about the game charactors.
The game charactors will often not share the same attributes, or use a basew
For example:
A format that would allow me to do something like this:
#include "standardsettings.config"
//include other files which this file
//then changes
FastSpaceship:
Speed: 10 //pixels/sec
Rotation: 5 //deg/sec
MotherShip : FastSpaceship //inherits all the settings of the Spaceship ship
ShieldRecharge: 4
WeaponA [ power:10,
range:20,
style:fireball]
SlowMotherShip : MotherShip //inherits all the settings of the monther ship
Speed: 4 // override speed
I've been searching for a pre-existing format that does all this, or is similar, but with no luck. I'm keen not to reinvent the wheel unless I have to, so i was wondering if anyone knows any good configuration file formats that support these features
JSON is about the simplest file format around, has mature libraries, and you can interpret it to do anything you want.
{
"FastSpaceship" : {
"Speed" : 10,
"Rotation" : 5
},
"MotherShip" : {
"Inherits" : "FastSpaceship",
"ShieldRecharge" : 4,
"WeaponA": {
"Power": 10,
"Range": 20,
"style": "fireball"
}
},
"SlowMotherShip": {
"Inherits": "MotherShip",
"Speed": 4
}
}
YAML? It's like JSON without the commas and quotes.
You might want to check out some sort of frame-based representation, since it seems that's exactly what you're talking about. That wikipedia page links to a few existing implementations that maybe you could use, or create your own.
After alot of searching i've found a pretty good solution using Lua
Lua I found out was originally designed as a configuration file language, but then evolved into a complete programming language.
Example
util.lua
-- helper function needed for inheritance
function inherit(t) -- return a deep copy (incudes all subtables) of the table t
local new = {} -- create a new table
local i, v = next(t, nil) -- i is an index of t, v = t[i]
while i do
if type(v)=="table" then v=inherit(v) end -- deep copy
new[i] = v
i, v = next(t, i) -- get next index
end
return new
end
globalsettings.lua
require "util"
SpaceShip = {
speed = 1,
rotation =1
}
myspaceship.lua
require "globalsettings" -- include file
FastSpaceship = inherits(SpaceShip)
FastSpaceship.Speed = 10
FastSpaceship.Rotation = 5
MotherShip = inherits(FastSpaceship)
MotherShip.ShieldRecharge = 4
ShieldRecharge.WeaponA = {
Power = 10,
Range = 20,
Style = "fireball"
SlowMotherShip = inherits(MotherShip)
SlowMotherShip.Speed = 4
Using the print function in Lua its also easy to test the settings if they are correct. Syntax isn't quite as nice as I would like it, but its so close to what I want, i'm not gonna mind writing out a bit more.
The using the code here http://windrealm.com/tutorials/reading-a-lua-configuration-file-from-c.php I can read the settings into my C++ program