" I'm new in neural networks and DL4j, and I want to train neural network with CSV and build linear regression. How can I fix these errors "Cannot resolve method'.iterations and getFeatureMatrix()'"?
"Previously I'm tried to do that, but have another error in 'seed'".
import org.datavec.api.records.reader.RecordReader;
import org.datavec.api.records.reader.impl.csv.CSVRecordReader;
import org.datavec.api.split.FileSplit;
import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator;
import org.deeplearning4j.nn.api.OptimizationAlgorithm;
import org.deeplearning4j.nn.conf.MultiLayerConfiguration;
import org.deeplearning4j.nn.conf.NeuralNetConfiguration;
import org.deeplearning4j.nn.conf.Updater;
import org.deeplearning4j.nn.conf.layers.DenseLayer;
import org.deeplearning4j.nn.conf.layers.OutputLayer;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.deeplearning4j.nn.weights.WeightInit;
import org.deeplearning4j.optimize.listeners.ScoreIterationListener;
import org.nd4j.evaluation.classification.Evaluation;
import org.nd4j.linalg.activations.Activation;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.api.DataSet;
import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
import org.nd4j.linalg.lossfunctions.LossFunctions;
import java.io.File;
public class Data {
public static void main(String[] args) throws Exception {
Parameters:
int seed = 3000;
int batchSize = 200;
double learningRate = 0.001;
int nEpochs = 150;
int numInputs = 2;
int numOutputs = 2;
int numHiddenNodes = 100;
Load data:
//load data train
RecordReader rr = new CSVRecordReader();
rr.initialize(new FileSplit(new File("train.csv")));
DataSetIterator trainIter = new RecordReaderDataSetIterator(rr, batchSize, 0, 2);
//load test data
RecordReader rrTest = new CSVRecordReader();
rr.initialize(new FileSplit(new File("test.csv")));
DataSetIterator testIter = new RecordReaderDataSetIterator(rrTest, batchSize, 0, 2);
Network Configuration:
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(seed)
.iterations(1000)
.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
.learningRate(learningRate)
.updater(Updater.NESTEROVS).momentum(0.9)
.list()
.layer(0, new DenseLayer.Builder()
.nIn(numInputs)
.nOut(numHiddenNodes)
.weightInit(WeightInit.XAVIER)
.activation(Activation.fromString("relu"))
.build())
.layer(1, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
.weightInit(WeightInit.XAVIER)
.activation(Activation.fromString("softmax"))
.weightInit(WeightInit.XAVIER)
.nIn(numHiddenNodes)
.nOut(numOutputs)
.build()
)
.pretrain(false).backprop(true).build();
MultiLayerNetwork model = new MultiLayerNetwork(conf);
model.init();
model.setListeners(new ScoreIterationListener((15)));
for (int n = 0; n < nEpochs; n++) {
model.fit((trainIter));
System.out.println(("--------------eval model"));
Evaluation eval = new Evaluation(numOutputs);
while (testIter.hasNext()) {
DataSet t = testIter.next();
INDArray features = getFeatureMatrix();
INDArray lables = t.getLabels();
INDArray predicted = model.output(features, false);
eval.eval(lables, predicted);
}
System.out.println(eval.stats());
}
}
}
Logs
Build
First you should consider to use more class (like one for the definition of the neural network, one for the training process etc, ...). Just a best practice stuff.
I do not know which version of DL4J you're using but we can notice that getFeatureMatrix() has been removed. One more thing is that this function should be called on a DataSet object and not "statically" like you seem to do. (you should do t.getFeatureMatrix()).
It is pretty same things about iterations() function of the neural network creation; This function has been removed since some DL4J releases. You can get more information about this function on this thread. Now you have to find an alternative to set up number of iteration, you can take a look at this thread. Hope it is answering your question !
Related
I am trying to follow different papers and tutorials to learn how to solve optimization problems of modelica modells.
In http://www.syscop.de/files/2015ss/events/opcon-thermal-systems/optimization_tool_chain_in_jmodelica.org_toivo_henningsson.pdf I found a very simple tutorial. But when I execute it I get some very open error messages.
I am using Python 2.7 with jupyther.
Here is my Notepad:
from pyjmi import transfer_optimization_problem
import matplotlib.pyplot as plt
import os.path
file_path = os.path.join("D:\Studies", "Integrator.mop")
op = transfer_optimization_problem('optI', file_path)
res = op.optimize()
t = res['time']
x = res['x']
u = res['u']
plt.plot(t,x,t,u)
My modelica file:
package Integrator
model Integrator
Real x(start=2, fixed = true);
input Real u;
equation
der(x) = -u;
end Integrator;
optimization optI(objective = finalTime, objectiveIntegrand = x^2 + u^2, startTime = 0, finalTime(free = true, min = 0.5, max = 2, initialGuess = 1))
Real x (start = 2, fixed = true);
input Real u;
equation
der(x) = -u;
constraint
u <= 2;
x(finalTime) = 0;
end optI;
end Integrator;
When I excute the code I get an RuntimeError, telling me that a java error occured and details where printed. From the Traceback I do not know what the note
This file is compatible with both classic and new-style classes
mean. I know that my setup is working because I executed the CSTR tutorial given by modelon. But now, it try to use my own models and it is giving me that error.
Runtime Error desciption
Using same syntax like in Modelica for import
e.g.
import Modelica.SIunits.Temperature;
where the package structure is part of the model-identification should resolve the issue.
op = transfer_optimization_problem('Integrator.optI', file_path)
I want to plot a simple x,y curve using jfreechart using the following code :
package temp;
import java.awt.BasicStroke;
import java.awt.Color;
import java.awt.Font;
import javax.swing.BorderFactory;
import javax.swing.JFrame;
import javax.swing.SwingUtilities;
import java.util.List;
import org.jfree.chart.ChartFactory;
import org.jfree.chart.ChartPanel;
import org.jfree.chart.JFreeChart;
import org.jfree.chart.plot.PlotOrientation;
import org.jfree.chart.plot.XYPlot;
import org.jfree.chart.renderer.xy.StandardXYItemRenderer;
import org.jfree.data.xy.XYSeries;
import org.jfree.data.xy.XYSeriesCollection;
public class recorder extends JFrame {
int number=0;
public recorder(int number, List time, List values) {
this.number=number;
XYSeries series = new XYSeries(String.valueOf(number));
// The 4 following lines are slow
int n_value=time.size();
for (int i=0;i<n_value;i++){
series.add((double)time.get(i),(double)values.get(i));
}
// and that I would like to replace them by a "simple" assignement
XYSeriesCollection dataset = new XYSeriesCollection(series);
JFreeChart chart = ChartFactory.createXYLineChart("value","t","y",dataset,PlotOrientation.VERTICAL,true,true,false);
XYPlot plot = chart.getXYPlot();
StandardXYItemRenderer renderer = new StandardXYItemRenderer();
renderer.setSeriesPaint(0, Color.BLACK);
plot.setRenderer(renderer);
plot.setBackgroundPaint(Color.white);
plot.setRangeGridlinesVisible(false);
plot.setDomainGridlinesVisible(false);
ChartPanel chartPanel = new ChartPanel(chart);
chartPanel.setPreferredSize(new java.awt.Dimension(600, 400));
chartPanel.setBackground(Color.white);
setContentPane(chartPanel);
pack();
setTitle("recorder"+String.valueOf(number));
setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
}
}
Does anybody know a way to do this and to get rid of the loop and the step-by-step add() method ? Thanks in advance for your help.
Create a custom class that extends AbstractXYDataset, as shown here, and give it a constructor or public method that accepts a java.util.List<E>. Then, instead of iterating though the list redundantly, your implementations of getX() and getY() can simply return the required value.
i downloaded the last version of iTextSharp dll. I generated a PdfPTable object and i have to set it's height. Despite to set width of PdfPTable, im not able to set height of it. Some authors suggest to use 'setFixedHeight' method. But the last version of iTextSharp.dll has not method as 'setFixedHeight'. It's version is 5.5.2. How can i do it?
Setting a table's height doesn't make sense once you start thinking about it. Or, it makes sense but leaves many questions unanswered or unanswerable. For instance, if you set a two row table to a height of 500, does that mean that each cell get's 250 for a height? What if a large image gets put in the first row, should the table automatically respond by splitting 400/100? Then what about large content in both rows, should it squish those? Each of these scenarios produces different results that make knowing what a table will actually do unreliable. If you look at the HTML spec you'll see that they don't even allow setting a fixed height for tables.
However, there's a simple solution and that's just setting the fixed height of the cells themselves. As long as you aren't using new PdfPCell() you can just set DefaultCell.FixedHeight to whatever you want.
var t = new PdfPTable(2);
t.DefaultCell.FixedHeight = 100f;
t.AddCell("Hello");
t.AddCell("World");
t.AddCell("Hello");
t.AddCell("World");
doc.Add(t);
If you are manually creating cells then you need to set the FixedHeight on each:
var t = new PdfPTable(2);
for(var i=0;i<4;i++){
var c = new PdfPCell(new Phrase("Hello"));
c.FixedHeight = 75f;
t.AddCell(c);
}
doc.Add(t);
However, if you want normal table-ness and must set a fixed height that chops things that don't fit you can also use a ColumnText. I wouldn't recommend this but you might have a case for it. The code below will only show six rows.
var ct = new ColumnText(writer.DirectContent);
ct.SetSimpleColumn(100, 100, 200, 200);
var t = new PdfPTable(2);
for(var i=0;i<100;i++){
t.AddCell(i.ToString());
}
ct.AddElement(t);
ct.Go();
you can use any one of following
cell.MinimumHeight = 20f;
or
cell.FixedHeight = 30f;
The premise is that you have downloaded the jar iText jar,you can try this code,you can achive this function,In a A4 paper out a row of three columns of dataeg:
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import com.lowagie.text.Document;
import com.lowagie.text.DocumentException;
import com.lowagie.text.Paragraph;
import com.lowagie.text.pdf.PdfPCell;
import com.lowagie.text.pdf.PdfPTable;
import com.lowagie.text.pdf.PdfWriter;
public class ceshi {
public static final String DEST = "D:\\fixed_height_cell.pdf";
public static void main(String[] args) throws IOException,
DocumentException {
File file = new File(DEST);
file.getParentFile().mkdirs();
new ceshi().createPdf(DEST);
}
public void createPdf(String dest) throws IOException, DocumentException {
Document document = new Document();
PdfWriter.getInstance(document, new FileOutputStream(dest));
document.open();
PdfPTable table = new PdfPTable(3);// Set a row and the three column of
// A4 paper
table.setWidthPercentage(100);
PdfPCell cell;
for (int r = 1; r <= 2; r++) {// Set display two lines
for (int c = 1; c <= 3; c++) {// Set to display a row of three columns
cell = new PdfPCell();
cell.addElement(new Paragraph("test"));
cell.setFixedHeight(285);// Control the fixed height of each cell
table.addCell(cell);
}
}
document.add(table);
document.close();
}
}
I have to HDFS folder size which is having sub directories from java.
From command line we can use -dus option, But anyone can help me on how to get the same using java.
The getSpaceConsumed() function in the ContentSummary class will return the actual space the file/directory occupies in the cluster i.e. it takes into account the replication factor set for the cluster.
For instance, if the replication factor in the hadoop cluster is set to 3 and the directory size is 1.5GB, the getSpaceConsumed() function will return the value as 4.5GB.
Using getLength() function in the ContentSummary class will return you the actual file/directory size.
You could use getContentSummary(Path f) method provided by the class FileSystem. It returns a ContentSummary object on which the getSpaceConsumed() method can be called which will give you the size of directory in bytes.
Usage :
package org.myorg.hdfsdemo;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class GetDirSize {
/**
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
Configuration config = new Configuration();
config.addResource(new Path(
"/hadoop/projects/hadoop-1.0.4/conf/core-site.xml"));
config.addResource(new Path(
"/hadoop/projects/hadoop-1.0.4/conf/core-site.xml"));
FileSystem fs = FileSystem.get(config);
Path filenamePath = new Path("/inputdir");
System.out.println("SIZE OF THE HDFS DIRECTORY : " + fs.getContentSummary(filenamePath).getSpaceConsumed());
}
}
HTH
Thank you guys.
Scala version
package com.beloblotskiy.hdfsstats.model.hdfs
import java.nio.file.{Files => NioFiles, Paths => NioPaths}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
import org.apache.commons.io.IOUtils
import java.nio.file.{Files => NioFiles}
import java.nio.file.{Paths => NioPaths}
import com.beloblotskiy.hdfsstats.common.Settings
/**
* HDFS utilities
* #author v-abelablotski
*/
object HdfsOps {
private val conf = new Configuration()
conf.addResource(new Path(Settings.pathToCoreSiteXml))
conf.addResource(new Path(Settings.pathToHdfsSiteXml))
private val fs = FileSystem.get(conf)
/**
* Calculates disk usage with replication factor.
* If function returns 3G for folder with replication factor = 3, it means HDFS has 1G total files size multiplied by 3 copies space usage.
*/
def duWithReplication(path: String): Long = {
val fsPath = new Path(path);
fs.getContentSummary(fsPath).getSpaceConsumed()
}
/**
* Calculates disk usage without pay attention to replication factor.
* Result will be the same with hadopp fs -du /hdfs/path/to/directory
*/
def du(path: String): Long = {
val fsPath = new Path(path);
fs.getContentSummary(fsPath).getLength()
}
//...
}
Spark-shell tool to show all tables and their consumption
A typical and illustrative tool for Spark-shell, looping loop over all bases, tables and partitions, to get sizes and report into a CSV file:
// sshell -i script.scala > ls.csv
import org.apache.hadoop.fs.{FileSystem, Path}
def cutPath (thePath: String, toCut: Boolean = true) : String =
if (toCut) thePath.replaceAll("^.+/", "") else thePath
val warehouse = "/apps/hive/warehouse" // the Hive default location for all databases
val fs = FileSystem.get( sc.hadoopConfiguration )
println(s"base,table,partitions,bytes")
fs.listStatus( new Path(warehouse) ).foreach( x => {
val b = x.getPath.toString
fs.listStatus( new Path(b) ).foreach( x => {
val t = x.getPath.toString
var parts = 0; var size = 0L; // var size3 = 0L
fs.listStatus( new Path(t) ).foreach( x => {
// partition path is x.getPath.toString
val p_cont = fs.getContentSummary(x.getPath)
parts = parts + 1
size = size + p_cont.getLength
//size3 = size3 + p_cont.getSpaceConsumed
}) // t loop
println(s"${cutPath(b)},${cutPath(t)},${parts},${size}")
// display opt org.apache.commons.io.FileUtils.byteCountToDisplaySize(size)
}) // b loop
}) // warehouse loop
System.exit(0) // get out from spark-shell
PS: I checked, size3 is always 3*size, no extra information.
Below is a program that shows how strings are output incorrectly if the ARFF saver from weka is writing in incremental mode. The program below runs in incremental mode if a parameter is passed to the program and in batch mode if no parameter is passed.
Note that in batch mode, the ARFF file contains strings ... normal operation.
In incremental mode, the ARFF file contains integers in place of strings ... strange !
Any ideas on how to get the ARFF formater to output strings in incremental format?
import java.io.File;
import java.io.IOException;
import weka.core.Attribute;
import weka.core.FastVector;
import weka.core.Instance;
import weka.core.Instances;
import weka.core.converters.ArffSaver;
import weka.core.converters.Saver;
public class ArffTest {
static Instances instances;
static ArffSaver saver;
static boolean flag=false;
public static void addData(String ticker, double price) throws IOException{
int numAttr = instances.numAttributes(); // same for
double[] vals = new double[numAttr];
int i=0;
vals[i++] = instances.attribute(0).addStringValue(ticker);
vals[i++] = price;
Instance instance = new Instance(1.0, vals);
if (flag)
saver.writeIncremental(instance);
else
instances.add(instance);
}
public static void main(String[] args) {
if(args.length>0){
flag=true;
}
FastVector atts = new FastVector(); // attributes
atts.addElement(new Attribute("Ticker", (FastVector)null));// symbol
atts.addElement(new Attribute("Price")); // price that order exited at.
instances = new Instances("Samples", atts, 0); // create header
saver = new ArffSaver();
saver.setInstances(instances);
if(flag)
saver.setRetrieval(Saver.INCREMENTAL);
try{
saver.setFile(new File("test.arff"));
addData("YY", 23.0);
addData("XY", 24.0);
addData("XX", 29.0);
if(flag)
saver.writeIncremental(null);
else
saver.writeBatch();
}catch(Exception e){
System.out.println("Exception");
}
}
}
You forgot to add the newly created Instance to the dataset.
Instance instance = new DenseInstance(1.0, vals);
instance.setDataset(instances); //Add instance!
if (flag)
saver.writeIncremental(instance);
else
instances.add(instance);
The Instance must have access to the dataset to retrieve the String
attribute. If it doesn't it just writes out the index.
Besides that I recommend to use Weka 3.7.6. Instance is now an
interface with two implementations.
cheers,
Muki